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Abstract 

Over  the  last  12  years,  the  percentage  of  the  Department  of  Defense  (DoD)  budget 
spent  on  the  procurement  of  services  has  risen  consistently  (Gansler,  2001).  In  an 
attempt  to  maximize  cost  savings  in  the  rapidly  growing  services  sector,  the  DoD 
established  a  Performance  Based  Service  Acquisition  (PBSA)  strategy  that  focuses  on 
evaluating  contractor  performance  based  on  their  ability  to  meet  desired  outcomes  rather 
than  the  means  to  which  the  outcomes  are  obtained.  In  April  2000,  Dr.  Gansler,  then 
Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics,  issued  a 
memorandum  mandating  that  50  percent  of  all  eligible  service  acquisitions  be  awarded 
using  PBSA  methods  by  Fiscal  Year  (FY)  2005.  While  some  studies  (Ausnik,  Gamin,  & 
Cannon,  2001;  Ausnik,  Baldwin,  Hunter,  &  Shirley,  2002)  have  attempted  to  measure  the 
potential  increases  in  quality  and  cost  savings,  very  little  research  has  been  focused  on 
USAF  implementation  of  PBSA  and  the  progress  and  attainment  of  the  PBSA  goals. 

Using  multiple  years  of  comprehensive  data  obtained  from  the  Air  Force 
Contract  Reporting  System,  also  known  as  the  J001,  this  thesis  extends  on  previous 
PBSA  research  (Lacey,  2004)  and  seeks  to  evaluate  and  analyze  the  current  and  expected 
future  states  of  PBSA  implementation  in  the  USAF,  including  an  assessment  of  current 
performance  against  PBSA  goals,  the  development  of  forecasts  of  future  performance 
against  PBSA  goals,  and  the  evaluation  of  PBSA  contract  characteristics.  A  combination 
of  descriptive  statistics,  forecasting,  contingency  tables,  and  regression  were  used  to 
analyze  the  data,  draw  conclusions,  and  make  recommendations  for  PBSA 
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implementation  improvements.  The  results  conclude  that  the  USAF  is  not  meeting 
interim  PBSA  goals  and  will  most  likely  fall  short  of  the  FY  2005  PBSA  goal.  These 
results  suggest  that  the  goals  may  not  have  been  reasonable  and  that  the  USAF  has  hit  a 
natural  plateau  in  PBSA  use. 
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Performance  Based  Service  Acquisition:  A  Quantitative  Evaluation  of  Implementation 
Goals  and  Perfonnance  in  the  United  States  Air  Force 

I.  Introduction 


Overview 

The  use  of  Performance  Base  Service  Acquisition  (PBS A)  for  the  acquisition  of 
services  has  become  increasingly  prevalent  in  the  United  States  Air  Force  (USAF)  as  a 
result  of  PBS  A  goals.  These  goals  were  mandated  in  an  attempt  to  increase  service 
quality  and  garner  potential  cost  savings  associated  with  the  use  of  PBS  A.  While  some 
studies  (Ausnik,  Cainm,  &  Cannon,  2001;  Ausnik,  Baldwin,  Hunter,  &  Shirley,  2002) 
have  attempted  to  measure  these  potential  increases  in  quality  and  cost  savings,  very  little 
research  (for  one  such  study  see  Lacey,  2004)  has  been  focused  on  USAF  implementation 
of  PBS  A  and  the  progress  and  attainment  of  the  mandated  PBS  A  goals. 

Using  multiple  years  of  data,  this  thesis  extends  Lacey’s  research  (Lacey,  2004) 
and  seeks  to  evaluate  and  analyze  the  current  and  future  states  of  PBS  A  implementation 
in  the  USAF,  including  an  assessment  of  current  perfonnance  against  PBSA  goals,  the 
development  of  forecasts  of  future  performance  against  PBSA  goals,  and  the  evaluation 
of  PBSA  contract  characteristics.  Additionally,  this  thesis  will  provide  recommendations 
for  PBSA  implementation  improvements  in  order  to  assist  USAF  leadership  in  making 
decisions  based  on  the  current  state  of  PBSA. 
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Background 

Over  the  last  12  years,  the  percentage  of  the  Department  of  Defense  (DoD)  budget 
spent  on  the  procurement  of  services  has  risen  consistently  (Gansler,  2001).  In  an 
attempt  to  increase  cost  savings  in  this  rapidly  growing  procurement  sector,  the  DoD 
established  a  perfonnance  based  service  acquisition  strategy  that  focuses  on  evaluating 
contractor  performance  based  on  their  ability  to  meet  desired  outcomes  rather  than  the 
means  to  which  the  outcomes  are  obtained.  In  April  2000,  Dr.  Gansler,  then  Under 
Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics,  issued  a  memorandum 
mandating  that,  at  a  minimum,  50  percent  of  all  eligible  service  acquisitions,  measured  in 
both  dollars  and  actions,  be  awarded  using  PBSA  methods  by  Fiscal  Year  (FY)  2005. 

In  order  to  understand  the  potential  benefits  of  PBSA,  it  is  first  important  to 
understand  the  meaning  and  concepts  of  PBSA.  The  Guidebook  for  Performance-Based 
Services  Acquisitions  in  the  Department  of  Defense  (PBSA  Guidebook)  defines  PBSA  as 
acquisition  strategies,  methods,  and  techniques  that  describe  and  communicate 
measurable  outcomes  rather  than  direct  perfonnance  processes  (DoD,  2001).  Simply, 
PBSA  asks  contractors  to  meet  a  desired  outcome  rather  than  telling  them  how  to  meet 
the  outcome.  Federal  Acquisition  Regulation  (FAR)  Subpart  37.6  describes 
characteristics  consistent  with  PBSA  designated  contracts.  According  to  FAR  Subpart 
37.6,  perfonnance-based  contracts  or  task  orders  exhibit  the  following  characteristics: 

(1)  Describe  requirements  in  terms  of  results  required  rather  than  the  methods  of 
performance  of  the  work. 

(2)  Use  measurable  performance  standards  and  quality  assurance  surveillance 
plans. 
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(3)  Specify  procedures  for  reduction  of  fee  or  reduction  to  price  of  a  fixed  price 
contract  when  services  are  not  performed  or  do  not  meet  contract 
requirements. 

(4)  Include  performance  incentives  when  appropriate. 

Past  research  suggests  that  by  utilizing  these  characteristics  and  describing 
requirements  in  terms  of  perfonnance  outcomes,  beneficial  outcomes  or  objectives  will 
be  achieved.  For  instance,  in  1998  the  Office  of  Federal  Procurement  Policy  (OFPP), 
which  is  a  central  office  that  provides  direction  for  procurement  policy  and  the 
development  of  procurement  systems,  concluded  a  four  year  study  on  PBSA.  According 
to  the  study,  the  utilization  of  PBSA  reduced  contract  prices,  improved  customer 
satisfaction,  and  increased  competition  (OFPP,  1998).  The  PBSA  Guidebook  also 
suggests  potential  benefits  of  PBSA  usage  by  listing  the  five  objectives  that  the  use  of 
PBSA  can  help  achieve.  Quoting  from  the  Guidebook,  these  five  objectives  are: 

(1)  Maximize  Performance  -  By  following  its  own  best  practices,  a  contractor  can 
deliver  the  required  service.  Since  the  prime  focus  is  on  the  end  result, 
contractors  can  adjust  their  processes,  as  appropriate,  throughout  the  life  of 
the  contract  without  the  burden  of  contract  modifications.  This  adjustment 
may  be  done  as  long  as  the  delivered  service  (outcome)  remains  in  accordance 
with  the  contract.  The  use  of  incentives  further  motivates  contractors  to 
furnish  their  best  performance. 

(2)  Maximize  Competition  and  Innovation  -  Encouraging  innovation  from  the 
supplier  base  by  using  perfonnance  requirements  maximizes  opportunities  for 
competitive  alternatives  in  lieu  of  government-directed  solutions.  Since 
PBSA  allows  for  greater  innovation,  it  has  the  potential  to  attract  a  broader 
industry  base. 

(3)  Encourage  and  Promote  the  Use  of  Commercial  Services  -  The  vast  majority 
of  service  requirements  are  commercial  in  nature.  Use  of  Federal  Acquisition 
Regulation  (FAR)  Part  12  (Acquisition  of  Commercial  Items)  procedures 
provides  great  benefits  by  minimizing  the  reporting  burden  and  reducing  the 
use  of  government-unique  contract  clauses  and  similar  requirements,  which 
can  help  attract  a  broader  industry  base. 
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(4)  Shift  in  Risk  -  When  contractors  become  responsible  for  achieving  the 
objectives  in  the  work  statement  through  the  use  of  their  own  best  practices 
and  processes,  much  of  the  risk  is  shifted  from  the  government  to  industry. 
Agencies  should  consider  this  reality  in  detennining  the  appropriate 
acquisition  incentives. 

(5)  Achieve  Savings  -  Experience  in  both  government  and  industry  has 
demonstrated  the  use  of  performance  requirements  results  in  cost  savings. 

The  interest  in  applying  PBS  A  and  realizing  its  benefits  increases  as  the 
percentage  of  the  DoD  budget  spent  on  services  continues  to  grow.  In  1991  the  OFPP 
issued  the  first  policy  letter,  Policy  Letter  91-2,  addressing  PBS  A.  This  policy  letter 
established  policy  for  the  Government’s  acquisition  of  services  by  contract,  emphasizing 
the  use  of  performance  requirements  and  quality  standards  in  defining  contract 
requirements,  source  selection,  and  quality  assurance.  Since  the  issuance  of  Policy  Letter 
91-2,  multiple  directives  and  guidance  documents  have  been  issued  for  both  the  Federal 
Government  and  the  USAF.  However,  it  was  not  until  Dr.  Gansler’s  (2000) 
memorandum  on  PBSA  that  the  implementation  of  PBSA  in  the  USAF  began  to  take 
shape. 


Problem 

In  order  to  evaluate  the  perfonnance  of  PBSA  implementation,  USAF  decision 
makers  need  to  know  the  current  state  of  PBSA  in  the  USAF.  Furthennore,  USAF 
decision  makers  need  to  know  whether  or  not  mandated  PBSA  goals  are  currently  being 
met  and  will  be  met  in  the  future.  Additionally,  USAF  leadership  needs  to  know  if  these 
PBSA  goals  are  reasonably  attainable.  Recent  research  on  PBSA  implementation  in  the 
DoD  is  limited,  but  one  study  (Lacey,  2004)  suggests  that  the  USAF  is  not  on  course  to 
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meet  mandated  goals.  However,  Lacey’s  study  is  extremely  exploratory,  using  only  one 
year  of  USAF  comprehensive  data  for  analysis  (Lacey,  2004). 

Research  Question 

The  over  arching  research  question  is,  “What  are  the  current  and  future  expected 
states  of  PBSA  implementation  in  the  USAF?”  This  question  was  answered  by 
conducting  an  evaluation  and  analysis  of  the  current  and  future  states  of  PBSA 
implementation  in  the  USAF  using  archival  data  obtained  from  the  Air  Force  Contract 
Reporting  System,  also  known  as  the  J001.  The  evaluation  and  analysis  included  the 
assessment  of  current  performance  against  PBSA  goals,  forecasting  of  future 
performance  against  PBSA  goals,  the  evaluation  of  PBSA  contract  characteristics,  and 
recommendations  for  PBSA  implementation  improvement. 

Investigative  Questions 

Because  the  research  question  for  this  study  is  so  broad,  several  investigative 
questions  were  developed  in  an  attempt  to  evaluate  as  many  aspects  of  PBSA 
implementation  as  possible,  including  the  reliability  of  the  J001  database.  The  five 
investigative  questions  this  study  answers  are: 

IQ  1 .  Is  the  J001  database,  which  is  the  source  of  government  procurement  data, 
reliable? 

IQ2.  Is  the  USAF  meeting  interim  PBSA  goals? 

IQ3.  Is  the  USAF  on  track  to  meet  future  mandated  PBSA  goals? 

IQ4.  Is  the  percentage  of  modifications  coded  PBSA  equal  to  the  percentage  of 
non-modifications  coded  PBSA? 
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IQ5.  Are  any  particular  acquisition  characteristics  associated  with  PBS  A  use? 

IQ5a.  What  are  the  antecedents  to  the  use  of  PBSA? 

IQ5b.  What  are  the  barriers  to  the  use  of  PBSA? 

Investigative  question  one  was  developed  in  order  to  ensure  that  the  database  from  which 
the  data  for  this  study  was  extracted  was  reliable.  Investigative  questions  two  and  three 
were  developed  in  order  to  answer  the  obvious  questions  of  whether  or  not  the  USAF  is 
meeting  interim  and  will  meet  future  PBSA  goals.  Investigative  question  four  was 
developed  in  order  to  see  if  PBSA  percentages  are  consistent  for  new  contract  awards  and 
modifications.  This  is  important  because  a  difference  in  the  two  percentages  may  imply 
that  PBSA  contracts  are  modified  more  or  less  often  than  non-PBSA  contracts  or  that 
PBSA  modifications  are  more  or  less  costly  than  non-PBSA  modifications.  Lastly, 
investigative  question  five  was  developed  in  order  to  identify  contract  variables  that  may 
increase  or  decrease  the  probability  of  PBSA  use. 

Methodology 

The  first  investigative  question  was  formulated  in  order  to  ensure  reliability  of  the 
data  manually  entered  into  the  J001  database.  The  J001  is  a  database  comprised  of  DD 
Form  350,  Individual  Contracting  Action  Report,  infonnation  for  all  Air  Force 
contracting  actions  exceeding  $25,000.  Investigative  question  1  was  answered  by  taking 
a  sample  of  contracts  coded  PBSA  in  the  J001  database  and  evaluating  them  against  the 
four  mandatory  criteria  for  PBSA  identified  in  the  PBSA  Guidebook. 
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Investigative  questions  2  through  5  were  answered  by  collecting  data  directly 
from  the  J001,  for  all  USAF  service  contracts  from  FY  2001  through  FY  2004.  The  data 
was  analyzed  using  descriptive  statistics,  forecasting,  contingency  tables,  and  regression. 

Proposed  Contributions 

By  following  the  steps  outlined  above,  this  thesis  will  further  the  knowledge  and 
understanding  of  the  use  of  PBS  A  in  the  USAF.  First,  it  will  provide  insight  to  the 
reliability  of  the  J001  database.  Secondly,  it  will  allow  USAF  leadership  to  see  how  well 
the  USAF  is  meeting  interim  mandated  PBS  A  goals  and  progressing  towards  future 
goals.  Additionally,  this  study  will  validate  the  establishment  of  mandated  PBSA  goals 
and  help  decision-makers  establish  better  future  goals.  Lastly,  the  results  from  this  study 
will  help  procurement  personnel  identify  when  to  use  PBSA. 

Summary 

This  chapter  presented  the  overall  basic  components  of  the  research  effort.  First, 
the  background  to  PBSA  implementation  in  the  USAF  and  how  it  has  led  to  the 
formation  of  the  problem  statement,  research  questions,  and  subsequent  investigative 
questions  were  explained.  Next,  a  summary  of  current  knowledge  and  methodology  used 
to  answer  the  investigative  questions  was  presented.  Lastly,  the  proposed  contributions 
were  addressed.  The  following  chapter  will  provide  a  review  of  the  literature  relevant  to 
PBSA,  describing  the  evolution  of  PBSA  within  the  Federal  Government  and  the  DoD. 
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II.  Literature  Review 


Introduction 

This  chapter  reviews  the  literature  relevant  to  the  creation,  implementation,  and 
evaluation  of  the  Performance  Based  Service  Acquisition  (PBSA)  initiative  in  the  United 
States  Air  Force  (USAF).  Besides  basic  definitions  and  an  explanation  of  PBSA,  this 
chapter  explains  how  past  policy,  directives,  and  regulations  established  by  the  Federal 
Government,  Department  of  Defense  (DoD),  and  the  USAF  have  led  to  the  current  state 
of  PBSA  within  the  USAF  acquisition  and  contracting  community. 

Federal  Guidance 

Federal  Acquisition  Regulation  (FAR)  Part  37  provides  the  policy  and  guidance 
for  acquisition  and  management  of  service  contracts.  Although  the  FAR  does  not 
establish  any  objective  PBSA  goals,  its  contents  do  state  that  performance-based 
contracting  is  the  preferred  method  of  acquiring  services  and  requires  the  use  of  it  to  the 
maximum  extent  practicable. 

FAR  Subpart  37.101  defines  a  service  contract  as,  “a  contract  that  directly 
engages  the  time  and  effort  of  a  contractor  whose  primary  purpose  is  to  perfonn  an 
identifiable  task  rather  than  to  furnish  an  end  item  of  supply  (FAR  Part  37).” 

Additionally,  the  regulation  notes  that  services  can  be  performed  by  either  professional  or 
nonprofessional  personnel  and  on  an  individual  or  organizational  basis.  FAR  Subpart 
37.101  identifies  the  following  disciplines  where  service  contracts  may  be  found: 
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(1)  Maintenance,  overhaul,  repair,  servicing,  rehabilitation,  salvage, 

modernization,  or  modification  of  supplies,  systems,  or  equipment. 

(2)  Routine  recurring  maintenance  of  real  property. 

(3)  Housekeeping  and  base  services. 

(4)  Advisory  and  assistance  services. 

(5)  Operation  of  Government-owned  equipment  facilities,  and  systems. 

(6)  Communications  services. 

(7)  Architect-Engineering  (see  Subpart  36.6). 

(8)  Transportation  and  related  services  (see  Part  47). 

(9)  Research  and  development  (see  Part  35). 

It  is  worthy  to  note  that,  although  included  in  the  general  list  of  services,  research 
and  development  services  (FAR  Part  35),  architect-engineering  services  (FAR  Part  36), 
and  transportation  services  (FAR  Part  47)  are  governed  by  their  respective  FAR  Parts  in 
the  event  of  inconsistencies  in  the  FAR  (FAR  Part  37).  This  precedence  issue  can 
become  particularly  important  in  any  attempt  to  detennine  if  PBS  A  should  be  used  for 
the  acquisition  of  these  services. 

FAR  Subpart  37.102  excludes  specific  service  type  contracts,  including  architect¬ 
engineering  services,  construction,  utility  services,  and  services  that  are  identical  to 
supply  purchases,  from  performance-based  contracts.  In  addition,  FAR  Subpart  37. 102 
establishes  an  order  of  precedence  for  contract  types  when  acquiring  services.  The  order 
of  precedence  established  by  FAR  Subpart  37. 102  for  all  service  contracts  is: 

(1)  A  firm -fixed  price  performance-based  contract  or  task  order. 

(2)  A  perfonnance-based  contract  or  task  order  that  is  not  firm-fixed  price. 

(3)  A  contract  or  task  order  that  is  not  performance-based. 

This  order  of  precedence  suggests  that  the  DoD’s  top  priority  is  to  award  service 
contracts  or  task  orders  using  PBSA  and  preferably  using  a  FFP  contract.  The  next 
priority  suggested  from  this  order  of  precedence  is  to  use  PBSA  with  non-fixed  price 
contracts,  such  as  cost  reimbursement  contracts.  The  “last  resort”  is  to  award  contracts  or 
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task  orders  that  are  not  performance-based.  Although  this  order  of  precedence  provides 
only  generalized  guidance  for  the  use  of  PBS  A,  it  promotes  the  use  of  firm-fixed  price 
(FFP)  type  contracts  for  PBSA.  However,  by  identifying  the  use  of  perfonnance-based 
contracts  or  task  orders  that  are  not  FFP  as  a  possible  service  contract  option  it  can  be 
inferred  that  not  all  PBSA  contracts  need  to  or  can  be  FFP. 

FAR  Subpart  37.6  further  promotes  the  use  of  FFP  contracts  for  “services  that  can 
be  defined  objectively  and  for  which  the  risk  of  performance  is  manageable  (FAR  Part 
37).”  This  FAR  Subpart  also  prescribes  the  policies  and  procedures  for  the  use  of  PBSA, 
including  but  not  limited  to,  the  proper  preparation  of  a  PBSA  Statement  of  Work 
(SOW),  quality  assurance,  and  the  use  of  positive  and  negative  perfonnance  incentives. 
Furthermore,  according  to  FAR  Subpart  37.6,  PBSA  contracts: 

(1)  Describe  requirements  in  terms  of  results  required  rather  than  the  methods  of 
performance  of  the  work. 

(2)  Use  measurable  performance  standards  and  quality  assurance  surveillance 
plans. 

(3)  Specify  procedures  for  reduction  of  fee  or  reduction  to  price  of  a  fixed  price 
contract  when  services  are  not  performed  or  do  not  meet  contract 
requirements. 

(4)  Include  perfonnance  incentives  when  appropriate. 

In  1974,  Public  Law  93-400  required  the  Office  of  Management  and  Budget, 
through  its  Office  of  Federal  Procurement  Policy  (OFPP),  to  establish  a  system  for 
collecting,  developing,  and  disseminating  procurement  data,  which  took  into  account  the 
needs  of  Congress,  the  Executive  Branch,  and  the  public  (United  States  Congress,  1974). 
This  law  led  to  the  creation  of  the  Federal  Procurement  Data  System  (FPDS),  now  a 
requirement  under  FAR  Subpart  4.6.  The  FPDS  provides  a  comprehensive  mechanism 
for  assembling,  organizing,  and  presenting  contract  placement  data  for  the  Federal 
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Government.  Subsequently,  the  DoD  created  the  Defense  Contract  Action  Data  System 

(DC ADS)  which  transmits  all  DoD  procurement  data  to  the  FPDS.  In  order  to  collect  the 

appropriate  data  for  the  DCADS,  the  DoD  created  the  DD  Form  350,  Individual 

Contracting  Action  Report,  which  is  required  for  all  delivery/task  orders  exceeding 

$25,000.  The  DD  Form  350  includes  infonnation  about  not  only  the  type  of  action,  but 

also  numerous  other  contractor  socio-economic  characteristics,  such  as  business  size  and 

ethnic  ownership.  The  USAF  collects  and  transmits  all  DD  Form  350  information  to 

DCADS  via  the  Air  Force  Contract  Reporting  System,  also  known  as  the  J001.  The  DD 

Form  350  and  the  related  procurement  databases  are  valuable  sources  for  information 

pertaining  to  how  the  DoD,  and  specifically  the  USAF,  spends  taxpayer  money. 

In  1991  the  OFPP  issued  the  first  policy  letter  directed  towards  PBS  A,  Policy 

Letter  91-2.  This  letter  established  policy  for  the  Government’s  acquisition  of  services 

by  contract,  emphasizing  the  use  of  performance  requirements  and  quality  standards  in 

defining  contract  requirements,  source  selection,  and  quality  assurance.  The  letter  states, 

It  is  the  policy  of  the  Federal  Government  that  (1)  agencies  use  performance- 
based  contracting  methods  to  the  maximum  extent  practicable  when  acquiring 
services,  and  (2)  agencies  carefully  select  acquisition  and  contract  administration 
strategies,  methods,  and  techniques  that  best  accommodate  the  requirements 
(OFPP,  1991). 

Once  again,  the  preference  for  the  use  of  FFP  contracts  rather  than  cost  reimbursement 
contracts  is  emphasized  for  instances  where  services  can  be  objectively  defined  and  risk 
is  manageable.  However,  the  policy  letter  also  clearly  states  that  all  contracts,  regardless 
of  contract  type,  shall  include  incentive  provisions  to  ensure  that  contractors  are  rewarded 
for  good  performance  and  penalized  for  unsatisfactory  performance.  This  provision  adds 
to  the  more  basic  requirements  of  FAR  Subpart  37.6,  which  never  mandates  the  use  of 
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fee  or  price  reductions  for  unsatisfactory  perfonnance,  but  simply  states  to,  “include 
performance  incentives  when  appropriate  (FAR  Subpart  37.6).” 

In  the  infancy  of  PBS  A  implementation,  some  agencies  began  to  rely  on 
contractors  to  perform  certain  functions  in  such  a  way  as  to  raise  questions  about  who 
was  creating  Government  policy,  the  Government  or  private  contractors  (OFPP,  1992). 

In  addition,  the  amount  of  control  over  contract  performance  being  transferred  to 
contractors  began  to  be  questioned.  In  light  of  this,  the  OFPP  issued  Policy  Letter  92-1, 
establishing  Executive  Branch  policy  relating  to  service  contracting  and  inherently 
governmental  functions  in  order  to  avoid  unacceptable  transfer  of  official  responsibility 
to  Government  contractors.  Policy  92-1  specifically  defined  an  inherently  governmental 
function  as: 

[A]  function  that  is  so  intimately  related  to  the  public  interest  as  to  mandate 
performance  by  Government  employees.  These  functions  include  those  activities 
that  require  either  the  exercise  of  discretion  in  applying  Government  authority  or 
the  making  of  value  judgments  in  making  decisions  for  the  Government.  (OFPP, 
1992) 

The  policy  letter  noted  that  while  certain  functions,  such  as  facility  maintenance  and  food 
services,  may  be  performed  by  contractors,  other  functions,  such  as  the  command  of 
combat  troops  may  not.  However,  the  policy  letter  made  a  point  not  to  specify  legally 
which  functions  are  inherently  governmental  or  define  the  factors  used  to  make  such  a 
determination  by  using  non-binding  terminology,  such  as  “may”  or  “might.”  This 
intentional  lack  of  specifics  in  identifying  inherently  governmental  functions  may  cause 
internal  debate  over  the  use  of  contractors  to  perform  non-standard  services,  but  it  also 
allows  governmental  agencies  the  flexibility  to  use  creative  contracting  methods, 
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including  the  use  of  PBS  A,  when  acquiring  services  in  order  to  meet  mission 
requirements. 

A  1993  Office  of  Management  and  Budget  (OMB)  directed  agency  review 
revealed  that  service  contracting  practices  and  capabilities  were  uneven  across  the 
Executive  branch  (OFPP,  1994).  In  response,  the  OFPP  issued  Policy  Letter  93-1.  This 
policy  letter  established  Government-wide  policy,  assigned  responsibilities,  and  provided 
guiding  principles  for  Executive  Departments  and  agencies  in  managing  the  acquisition 
and  use  of  services.  In  addition,  the  policy  letter  uses  the  “best  practices”  concept  to 
guide  government  agencies  towards  the  use  of  a  more  results-oriented  approach  to  service 
contracting.  This  “best  practices”  concept  ultimately  led  to  the  creation  of  FAR  Subpart 
37.5,  which  allowed  contracting  officials,  for  the  first  time,  legally  to  use  practical 
techniques  gained  from  experience  to  improve  the  procurement  process  (FAR  Subpart 
37.5). 

In  a  further  attempt  to  encourage  the  use  of  PBS  A  and  a  more  results-oriented 
approach,  in  1997  the  OFPP  distributed  a  PBS  A  checklist  in  order  to  aid  in  developing 
performance-based  solicitations,  contracts  or  task  orders,  and  to  assist  in  detennining 
whether  existing  solicitations,  contracts,  or  task  orders  may  be  appropriately  classified  as 
performance-based  (OFPP,  1997).  The  checklist  provides  the  minimum  elements 
required  for  an  acquisition  to  be  considered  PBSA.  Once  again,  the  OFPP  purposefully 
made  the  checklist  vague  and  open  to  interpretation  in  order  to  avoid  infringing  on  the 
authority  or  discretion  of  contracting  officers.  Like  FAR  Subpart  37.6,  the  checklist 
identifies  four  minimum  requirements  for  PBSA.  These  requirements  are: 
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(1)  Performance  requirements  that  define  the  work  in  measurable  and  mission- 
related  terms. 

(2)  Perfonnance  standards  tied  to  the  performance  requirements. 

(3)  A  Government  quality  assurance  plan  that  describes  how  the  contractor’s 
performance  will  be  measured  against  the  perfonnance  standards. 

(4)  If  the  acquisition  is  either  critical  to  agency  mission  accomplishment  or 
requires  relatively  large  expenditures  of  funds,  positive  and  negative 
incentives  tied  to  quality  assurance  plan  measurements. 

Although  not  exactly  the  same,  these  requirements  are  similar  to  the  four  PBS  A  criteria 

identified  by  FAR  Subpart  37.6.  Furthermore,  the  instructions  within  the  checklist  assert 

that  the  checklist  is  only  one  of  many  tools  available  to  assist  in  the  developing  of  PBS  A 

contracts. 

In  1998  the  OFPP  concluded  a  four  year  study  of  PBS  A  and  published  a  report  of 
its  findings  titled,  “A  Report  on  the  Performance-Based  Service  Contracting  Pilot 
Project.”  The  study  included  twenty-six  non-PBSA  contracts,  ranging  from  $100,000  to 
$325  million.  These  contracts,  all  of  which  were  due  to  expire,  were  resolicited  using 
PBS  A  methods.  Before-and-after  measurements  were  taken  and  the  results  clearly 
demonstrated  PBSA’s  benefits  (OFPP,  1998).  According  to  the  report,  PBSA  reduced 
contract  prices  an  average  of  15%  at  all  price  ranges  and  across  all  types  of  services. 
More  importantly,  PBSA  was  particularly  effective  when  cost-reimbursement  non-PBSA 
contracts  were  converted  to  FFP  PBSA  contracts.  In  the  forward  of  the  report  Mr. 
Franklin  Raines,  OMB  Director,  encouraged  agency  officials  to  expand  the  use  of  and 
actively  promote  PBSA  in  order  to  capture  the  potential  billions  of  dollars  in  savings 
(OFPP,  1998).  Although  this  was  yet  another  attempt  to  encourage  the  use  of  PBSA,  it 
was  the  first  influential  directive  supported  by  factual  dollar  savings  data. 
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In  the  wake  of  the  PBS  A  pilot  project  and  its  significant  findings,  the  OFPP 
rushed  to  issue  the  “Guide  to  Best  Practices  for  Performance-Based  Service  Contracting.” 
Using  the  results  from  the  PBS  A  pilot  study  as  a  foundation,  the  document  contains 
suggested  best  practices  derived  from  the  experiences  of  contracting  personnel  in  both  the 
government  and  commercial  sector.  The  document  is  not  a  mandatory  regulation  nor  is  it 
a  “how  to”  manual,  but  it  is  a  tool  to  assist  in  developing  policies  and  procedures  for 
implementing  PBSA  (OFPP,  October  1998). 

For  the  next  few  years,  the  Federal  Government,  outside  of  the  DoD,  provided 
little  documented  PBSA  direction  until  March  of  2001,  when  then  OMB  Deputy  Director, 
Sean  O’Keefe,  issued  a  memorandum  to  all  department  heads  and  agencies  urging  them 
to  make  greater  use  of  PBSA  contracts.  This  memorandum  established  a  goal  for  FY 
2002  that,  for  all  contract  amounts  over  $25,000,  no  less  than  20  percent  of  the  total 
eligible  service  contracting  dollars  be  awarded  using  PBSA  techniques  (O’Keefe,  2001). 
This  marked  the  first  officially  documented  PBSA  goal  established  by  the  Federal 
Government,  outside  of  the  DoD. 

The  first  documentation  to  require  by  law  that  the  DoD  meet  a  goal  of  50%  PBSA 
of  services  by  FY  2005  was  the  National  Defense  Authorization  Act  (NDAA)  for  FY 
2002,  Sections  801-803.  Section  801  of  the  NDAA  covers  management  of  the 
procurement  of  services  and  includes  the  requirement  for  data  collection;  including 
service  purchased,  total  dollars,  type  of  contract,  business  size,  and  the  extent  of 
competition;  Section  802  establishes  DoD  PBSA  performance  goals  (United  States 
Congress,  2002).  This  section  specifies  the  following  minimum  usage  of  performance 
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based  purchasing  using  firm  fixed  prices  for  specific  tasks,  calculated  on  the  basis  of 
dollar  value: 

For  FY  2003,  a  percentage  no  less  than  25  percent 
For  FY  2004,  a  percentage  no  less  than  35  percent 
For  FY  2005,  a  percentage  no  less  than  50  percent 
For  FY  201 1,  a  percentage  no  less  than  70  percent 

Lastly,  Section  803  discusses  the  use  of  competition  in  all  multiple  award 

contracts  above  $100,000,  reemphasizing  the  need  for  competition  in  the  awarding  of 

service  contracts,  specifically  PBSA  contracts. 

In  response  to  sections  801  through  803  of  the  NDAA  for  FY  2002,  which 

established  a  series  of  requirements  impacting  the  acquisition  of  services  in  the 

Department  of  Defense,  Under  Secretary  of  Defense  for  Acquisition,  Technology  and 

Logistics,  Mr.  Edward  Aldridge,  issued  a  memorandum  to  all  secretaries  of  the  military 

departments  providing  policy  guidance  and  establishing  a  review  structure  and  process 

for  the  acquisition  of  services  in  accordance  with  section  801(d)  (Aldridge,  2002).  The 

attachments  to  this  memorandum  include  a  review  of  Department  of  Defense  acquisition 

of  services,  which  is  intended  to  outline  the  review  policy  for  the  acquisition  of  services 

and  ensure  service  acquisitions  are  based  on  clear,  performance-based  requirements,  that 

required  outcomes  are  identified  and  measurable,  and  that  the  acquisitions  are  properly 

planned  and  administered  to  achieve  intended  results. 

Realizing  that  agencies  were  making  only  moderate  progress  toward  PBSA 

implementation,  the  OFPP  established  a  PBSA  interagency  working  group  and  published 

the  results  and  finding  in  a  July  2003  report  titled,  “Interagency  Task  Force  on 

Performance-Based  Service  Acquisition.”  The  group  was  established  in  order  to  obtain  a 
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broader  understanding  of  the  requirements  of  PBS  A  and  to  identify  ways  to  increase 
PBSA  usage  (OFPP,  2003).  As  a  result  of  their  findings  the  group  recommended 
modifying  the  FAR  in  order  to  increase  flexibility  in  applying  PBSA,  modifying 
reporting  requirements  to  ensure  appropriate  PBSA  application,  and  improving  the 
quality,  currency,  and  availability  of  PBSA  guidance.  Although  acting  Under  Secretary 
of  Defense  for  Acquisition,  Technology  and  Logistics,  Mr.  Michael  Wynne  did  address 
these  recommendations  in  an  August  2003  memorandum,  but  as  of  March,  2005,  none  of 
the  recommended  modifications  to  the  FAR  have  been  published  (Wynne,  2003). 
However,  changes  in  reporting  requirements  and  an  increase  in  quality,  currency,  and 
availability  of  PBSA  guidance  has  been  implemented  through  the  use  of  web-based 
technology. 

While  the  OFPP  and  other  federal  government  agencies  were  attempting  to 
encourage  and  provide  guidance  for  the  use  of  PBSA  within  the  government,  the  DoD 
and  USAF  were  busy  trying  to  align  their  directives  and  instructions  with  federal 
guidance  in  order  to  meet  mandated  PBSA  requirements.  These  efforts  are  discussed  in 
the  following  section  of  the  chapter. 

Department  of  Defense  (DoD)  and  United  States  Air  Force  (USAF)  Guidance 

In  an  effort  to  promote  PBSA  within  the  Air  Force  acquisition  and  contracting 
community,  the  Principal  Deputy  Assistant  Secretary  of  the  Air  Force  for  Acquisition 
Management  published  Air  Force  Instruction  (AFI)  63-124  “Performance-Based  Service 
Contracts”  in  1999.  Upon  its  release,  AFI  63-124  replaced  AFM  64-108  “Service 
Contracts”  and  AFI  63-504  “Quality  Assurance  Evaluator  Program”  as  the  governing 
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guidance  for  service  contracting.  This  new  AFI  removed  numerous  mandated  processes, 
empowering  field  personnel  with  the  freedom  needed  to  embrace  agile  acquisition  and 
procurement  transfonnation  (DAF,  1999).  This  AFI  revolutionized  the  expectations  and 
methods  of  contracting  for  services  in  the  USAF  and  aligned  Air  Force  guidance  with  the 
PBSA  initiatives  established  by  the  OFPP.  This  AFI  was  later  amended  in  February  of 
2004  by  way  of  an  Interim  Change  to  AFI  63-124.  This  interim  change  deleted 
Attachment  2  of  AFI  63-124,  the  criteria  for  exemption  to  performance-based  service 
contracts,  leaving  only  the  services  listed  under  FAR  Part  37. 102  exempt  from  PBSA 
(DAF,  2004).  This  change  not  only  increases  the  number  of  services  eligible  for  PBSA, 
but  also  the  percentage  of  dollars.  At  first  glance,  it  seems  as  if  this  will  help  agencies 
meet  PBSA  goals,  when  in  reality,  it  may  make  achievement  of  PBSA  goals  more 
difficult  because  more  services  are  eligible,  thus  more  services  must  be  converted  to 
PBSA. 

In  April  2000,  the  Under  Secretary  of  Defense  of  Acquisition,  Technology,  and 
Logistics,  Dr.  Jacques  Gansler,  issued  an  official  memorandum  addressing  the  use  of 
PBSA  in  the  DoD.  In  this  memorandum  Dr.  Gansler  stated  that  the  use  of  performance- 
based  acquisition  strategies  for  services  was  among  his  highest  priorities  (Gansler,  2000). 
This  memorandum  mandated  that,  at  a  minimum,  50  percent  of  service  acquisitions, 
measured  in  both  dollars  and  actions,  be  PBSA  by  FY  2005.  From  the  memorandum  it  is 
unclear  whether  contract  modification  actions  for  service  contracts  should  be  used  for  the 
purpose  of  PBSA  percentage  calculations.  At  any  rate,  Dr.  Gansler’s  goal  was  set  forth 
nearly  one  year  before  any  other  federally  mandated  PBSA  goals,  suggesting  that  the 
DoD’s  implementation  of  PBSA  was  more  advanced  than  other  agencies.  In  addition  to 
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the  goals,  the  memorandum  outlines  policy  guidance,  implementation  planning,  training 
initiatives,  and  other  department-wide  PBSA  initiatives  in  order  to  help  guide  agencies 
towards  the  accomplishment  the  PBSA  goal. 

In  response  to  Dr.  Gansler’s  requirements,  the  USAF  issued  its  own  “PBSA 
Implementation  Plan.”  This  plan  provided  guidance  for  identification  of  services  eligible 
for  PBSA  including;  maintenance,  repair,  operations  and  support,  modifications, 
modernizations,  and  medical  services  (DAF,  2000).  More  importantly,  the 
implementation  plan  established  the  requirement  of  tracking  PBSA  implementation 
progress  through  automated  systems.  This  requirement  of  PBSA  tracking  is  what  allows 
researchers  the  opportunity  to  conduct  quantitative  studies  and  analysis  on  USAF 
implementation  of  PBSA. 

The  DoD  issued  The  Guidebook  for  Performance-Based  Services  Acquisitions  in 
the  Department  of  Defense  (PBSA  Guidebook)  in  2001  in  order  to  help  acquisition  teams 
better  understand  the  basic  principles  of  PBSA,  better  implement  performance-based 
strategies  and  methodologies  into  service  acquisitions,  and  meet  the  goals  set  forth  by  Dr. 
Gansler’s  2000  memorandum  (DAF,  2001).  Major  topics  include  market  research, 
developing  a  perfonnance-based  work  statement  and  establishing  measurable 
performance  standards,  incentives  and  remedies,  contractor  perfonnance  management, 
source  selection  considerations,  and  contract  administration.  The  PBSA  Guidebook  also 
identifies  the  following  minimum  elements  needed  in  order  for  an  acquisition  to  be 
considered  performance-based: 

(1)  Perfonnance  Work  Statement  -  The  perfonnance  work  statement  describes 
the  requirement  in  terms  of  measurable  outcomes  rather  than  by  means  of 
prescriptive  methods. 
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(2)  Measurable  Performance  Standards  -  To  determine  whether  performance 
outcomes  have  been  met,  measurable  performance  standards  define  what  is 
considered  acceptable  perfonnance. 

(3)  Remedies  -  Remedies  are  procedures  that  address  how  to  manage 
performance  that  does  not  meet  performance  standards.  While  not  mandatory, 
incentives  should  be  used,  where  appropriate,  to  encourage  perfonnance  that 
will  exceed  perfonnance  standards.  Remedies  and  incentives  complement 
each  other. 

(4)  Performance  Assessment  Plan  -  This  plan  describes  how  contractor 
performance  will  be  measured  and  assessed  against  performance  standards. 
(Quality  Assurance  Plan  or  Quality  Surveillance  Plan). 

As  shown  in  Table  1,  the  criteria  established  by  the  PBS  A  Guidebook,  although  not 

exactly  the  same,  parallel  PBSA  criteria  established  by  other  publications. 
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Table  1.  PBSA  Criteria  for  PBSA  Guidebook,  PBSA  Checklist,  and  FAR  Part  37.6 
PBS  A  Guidebook  PBSA  Checklist  FAR  Part  37.6 


1)  The  performance  work 
statement  describes  the 
requirement  in  terms  of 
measurable  outcomes  rather 
than  by  means  of 
prescriptive  methods. 


1)  Performance 
requirements  that  define 
the  work  in  measurable  and 
mission  related  tenns. 


1)  Describe  requirements 
in  terms  of  results  required 
rather  than  the  methods  of 
performance  of  the  work. 


2)  To  determine  whether 
performance  outcomes  have 
been  met,  measurable 
performance  standards 
define  what  is  considered 
acceptable  performance. 


2)  Perfonnance  standards 
tied  to  the  perfonnance 
requirements. 


2)  Use  measurable 
performance  standards  and 
quality  assurance 
surveillance  plans. 


3)  Remedies  are  procedures 
that  address  how  to  manage 
performance  that  does  not 
meet  performance  standards. 
While  not  mandatory, 
incentives  should  be  used, 
where  appropriate,  to 
encourage  performance  that 
will  exceed  performance 
standards.  Remedies  and 
incentives  complement  each 
other. 


3)  A  Government  quality 
assurance  plan  that 
describes  how  the 
contractor's  performance 
will  be  measured  against 
the  performance  standards 


3)  Specify  procedures  for 
reduction  of  fee  or 
reduction  to  price  of  a 
fixed  price  contract  when 
services  are  not  performed 
or  do  not  meet  contract 
requirements. 


4)  This  plan  describes  how 
contractor  performance  will 
be  measured  and  assessed 
against  perfonnance 
standards. 


4)  If  the  acquisition  is 
either  critical  to  agency 
mission  accomplishment  or 
requires  relatively  large 
expenditures  of  funds, 
positive  and  negative 
incentives  tied  to  quality 
assurance  plan 
measurements 


4)  Include  performance 
incentives  when 
appropriate. 


Lastly,  the  highlight  of  the  guidebook  is  the  “Top-Level  Guiding  Principles”  section  that 
summarizes  the  49-page  document  into  eleven  bullets,  emphasizing  perfonnance  based 
methods  and  incentives  based  on  well-defined  results-oriented  requirements. 
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In  early  2003,  the  USAF  issued  The  Management  and  Oversight  of  Acquisition  of 

Services  Process  (MOASP),  implementing  section  801  of  the  NDAA  for  FY  2002.  The 

guidance  appoints  the  Air  Force  Program  Executive  Officer  for  Services  (AFPEO/SV) 

(now  the  AFPEO  for  Combat  &  Mission  Support  (CM))  as  the  designated  official  to 

review  all  service  acquisitions,  except  major  weapon  system  and  space  program 

acquisitions  (DAF,  2003).  Additionally,  the  guidance  states  that  the  AFPEO/SV  may 

delegate  management  and  review  responsibilities  to  Major  Command  (MAJCOM) 

designated  officials,  which  then  may  further  delegate  these  authorities.  This  verbiage 

becomes  significant  when  determining  what  constitutes  a  perfonnance-based  service.  If 

such  determination  is  delegated  to  different  agencies  within  different  MAJCOMs, 

inconsistencies  in  PBSA  determinations  may  surface. 

In  August  2003,  a  memorandum  from  the  acting  Under  Secretary  of  Defense  for 

Acquisition,  Technology  and  Logistics,  Mr.  Michael  Wynne,  urged  the  Department  of 

Defense  to  continue  increasing  the  use  of  PBSA  because  such  acquisitions  provide 

significant  benefit  to  the  government  (Wynne,  2003).  Included  in  the  memorandum  are 

interim  goals  he  asks  each  military  department  to  work  towards  in  an  effort  to  meet  the 

goal  of  awarding  50  percent  of  all  contract  actions  and  dollars  using  performance-based 

specifications  by  FY  2005.  Mr.  Wynne’s  PBSA  goals  were; 

FY  2003  25  percent  of  dollars  awarded 

FY  2004  35  percent  of  dollars  awarded 

FY  2005  50  percent  of  dollars  awarded 

Although  these  goals  were  not  legislated,  they  do  offer  a  road  map  for  expected  PBSA 

progression  for  all  military  departments,  including  the  USAF. 
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Over  the  last  few  years  it  looked  as  if  the  DoD  was  well  on  its  way  to 
implementing  PBS  A  and  meeting  mandated  goals.  According  to  a  2002  GAO  testimony 
by  then  Acting  Director  of  Acquisition  and  Sourcing  Management,  Mr.  William  Woods, 
about  23  percent  of  eligible  service  contracts  were  reported  to  be  performance-based 
during  FY  2001  (Woods,  2002).  However,  in  August  2003,  Michael  Wynne,  claimed 
that  the  DoD  is  making  progress  towards  FY  2005  goal  of  50  percent  of  contract  actions 
awarded  using  performance-based  specifications  due  to  the  fact  that  in  FY  2002  over  20 
percent  of  the  service  requirements  were  awarded  using  performance-based  specifications 
(Wynne,  2003).  If  both  of  these  statements  are  correct,  it  shows  that  the  DoD  may  have 
made  minimal  progress  from  FY  2001  to  FY  2002,  but  still  seems  to  have  met  the  FY 
2003  goal  of  25  percent.  However,  recent  research  (Lacey,  2004)  suggests  that  the  FY 
2005  goal  of  50%  will  not  be  met  by  either  the  DoD  or  the  USAF. 

Other  Literature 

Previous  research  and  commercial  literature  on  PBSA  is  limited.  However,  the 
RAND  Corporation’s  Project  AIR  FORCE  division,  which  is  an  Air  Force  federally 
funded  research  and  development  center  for  studies  and  analysis,  has  conducted  a  few 
studies  directed  towards  implementation,  use,  and  effectiveness  of  PBSA. 

The  first  study,  titled,  “Performance-Based  Contracting  in  the  Air  Force:  A 
Report  on  Experiences  in  the  Field,”  was  completed  in  January  2001  and  looked  at 
examples  of  successful  USAF  applications  of  PBSA.  Twenty-two  recently  awarded 
contracts  from  “self-selected”  bases  that  identified  themselves  as  examples  of  successful 
implementation  of  PBSA  practices  were  studied.  The  results  concluded  that  most  bases 
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were  very  happy  with  the  performance  selected  using  PBSA  practices.  However,  only 
two  bases  attributed  the  success  in  obtaining  contractor  perfonnance  to  the  use  of  new 
acquisition  practices.  Additionally,  changes  in  costs  resulting  from  PBSA  practices  were 
difficult  to  determine  because  a)  many  of  the  new  contracts’  scope  of  work  was  different 
from  the  work  in  the  old  contracts,  b)  it  is  difficult  to  detennine  the  accuracy  of 
government  cost  estimates  associated  with  an  acquisition,  and  c)  it  is  difficult  to  measure 
changes  in  internal  costs,  such  as  costs  incurred  in  the  preparation  and  administration  of  a 
contract  using  PBSA.  (Ausnik,  Camm,  &  Cannon,  2001) 

The  second  study,  titled,  “Implementing  Performance-Based  Services  Acquisition 
(PBSA):  Perspectives  from  an  Air  Logistics  Center  and  a  Product  Center,”  focused  on  the 
application  of  PBSA  practices  at  program  offices  that  support  weapon  systems,  common 
subsystems,  and  special  mission  capabilities,  rather  than  operational  offices  that  mainly 
provide  installation  support  services.  Unlike  installation  support  services  that  are 
traditionally  commercial  in  nature  and  have  widely  accepted  performance  standards, 
many  services  purchased  by  program  offices  have  limited  opportunities  for  performance 
evaluation  and  detennination  of  successful  outcomes  because  many  of  the  desired  results 
of  a  service  are  not  always  known  in  advance.  The  study  found  that  Air  Logistic  Centers 
and  Product  Centers  are  having  difficulties  satisfying  the  PBSA  criteria  described  in  AFI 
63-124  for  the  use  of  “measurable  performance  standards”  because  they  interpret  the 
criteria  to  mean  that  a  desired  result  must  be  known  in  advance  in  order  to  measure 
performance.  However,  research  concluded  that  both  Centers  are  using  performance 
based  approaches  by  successfully  applying  the  other  three  criteria  described  in  AFI  63- 
124.  (Ausnik,  Baldwin,  Hunter,  &  Shirley,  2002) 
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Private  Sector  Literature 


Private  sector  literature  on  performance-based  service  contracts  is  limited.  This 
may  be  due  to  the  fact  that  in  the  private  sector,  only  5  to  20  percent  of  the  total  external 
procurement  budgets  are  used  to  procure  services  (Barry,  2003).  This  is  a  small 
percentage  considering  in  FY  1999,  the  dollar  amount  spent  by  the  DoD  on  services 
equaled  the  amount  spent  on  supplies/systems  (Gansler,  2001).  Despite,  the  limited 
emphasis  placed  on  private  sector  procurement  of  services,  some  guidance  on 
performance-based  contracts  outside  of  the  DoD  does  exist. 

In  an  Inside  Supply  Management  article  from  June  2003,  Jack  Barry  reveals  that’s 
there  is  a  private  sector  approach  to  building  perfonnance  based-contracts  that  has  been 
proven  successful  when  contracting  for  several  different  types  of  services,  including 
health/insurance,  transportation,  engineering,  advertising  and  legal  services.  The  article 
illustrates  that  an  effective  perfonnance-based  contract  must  identify  the  following  key 
factors  for  success: 

1)  Baseline  establishment 

2)  Escalators  and  audits 

3)  Incentives 

4)  Performance  measures 

Although  not  identical,  these  factors  for  success  are  similar  to  PBSA  criteria  identified 
for  government  procurement.  Baseline  establishment  involves  the  development  of  a 
range  of  acceptable  performance  that  is  based  on  market  research  and  historical  trends  in 
the  industry.  Performance  should  then  be  measured  against  these  baselines,  just  as 
government  PBSA  contracts  measure  perfonnance  against  established  performance 
standards  (Barry,  2004).  Private  incentives  are  similar  to  government  incentives  and 
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should  entice  the  supplier  to  reduce  costs.  Barry  notes  that  sharing  of  savings  tends  to  be 
a  strong  incentive  (Barry,  2004).  The  only  factor  that  truly  differs  from  government 
PBSA  guidance  is  escalators  and  audits.  Escalators  include  adjustment  clauses  that  allow 
baseline  prices  to  be  readjusted  upward  or  downward  according  to  predetermined  factors 
and  indices  (Barry,  2004).  Auditing  simply  entails  reviewing  documentation  associated 
with  adjustment  of  costs. 

In  2000,  a  commercial  advisory  firm,  named  Stqandish  Group,  traveled  to  four 
different  cities  and  hosted  a  total  of  14  workshops  in  an  attempt  to  explore  and  answer  the 
question,  “Will  perfonnance-based  contracts  increase  project  success  and  reduce  waste?” 
According  to  Stqandish  Group  forecasts,  commercial  firms  and  government  agencies 
wasted  over  $100  billion  in  consulting  fees  during  2000  (Johnson,  2000).  Although 
government  agencies  were  included  in  the  $100  billion  figure,  the  firm’s  study  focused 
specifically  on  the  commercial  sector  of  contracted  services.  While  the  study  was  unable 
to  substantiate  any  of  their  findings  quantitatively,  the  advisory  firm  did  publish  the 
information  gathered  from  workgroup  participants,  including  benefits  of  performance- 
based  contracts  and  how  to  implement  them.  According  to  their  research,  implementing 
a  performance-based  contract  is  a  five  step  process,  which  includes: 

Step  1:  Define  project  scope  and  objectives. 

Step  2:  Define  success  metrics. 

Step  3:  Establish  baselines. 

Step  4:  Measure  results. 

Step  5:  Implement  incentives  and  penalties. 

Once  again,  these  steps  are  very  similar  to  the  process  used  by  the  government  when 
establishing  PBSA  contracts,  hinting  that  the  same  methods  used  by  the  government  also 
apply  to  commercial  purchasing  practices. 
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Summary 

This  chapter  described  the  literature  relevant  to  the  creation,  implementation,  and 
evaluation  of  PBSA.  Besides  basic  definitions  and  explanations  of  PBSA,  this  chapter 
explained  how  past  policy,  directives,  and  regulations  established  by  the  Federal 
Government,  Department  of  Defense  (DoD),  and  the  USAF  have  led  to  the  current  state 
of  PBSA  within  the  USAF  acquisition  and  contracting  community. 
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ITT.  Methodology 


Introduction 

The  purpose  of  this  chapter  is  to  explain  the  research  methodology  used  to  answer 
the  question,  “What  are  the  current  and  expected  future  states  of  Performance  Based 
Service  Acquisition  (PBSA)  implementation  in  the  United  States  Air  Force  (USAF)?” 
The  majority  of  the  data  for  this  effort  were  extracted  from  the  Air  Force  Contract 
Reporting  System,  also  know  as  the  J001.  The  J001  is  a  database  comprised  of  DD  Form 
350,  Individual  Contracting  Action  Report,  inputs  for  all  Air  Force  contracting  actions 
exceeding  $25,000.  This  thesis  evaluates  and  analyzes  the  current  and  expected  future 
states  of  PBSA  implementation  in  the  USAF,  including  the  assessment  of  current 
performance  against  PBSA  goals,  the  forecasting  of  future  performance  against  PBSA 
goals,  and  the  evaluation  of  PBSA  contract  characteristics.  This  chapter  includes  the 
research  problem,  investigative  questions,  data  gathering,  data  reliability,  data  analysis, 
and  summary. 

Research  Problem 

Since  Dr.  Gansler,  then  Under  Secretary  of  Defense  for  Acquisition,  Technology 
and  Logistics,  mandated  that  50  percent  of  all  eligible  service  acquisitions,  measured  in 
both  dollars  and  actions,  be  awarded  using  PBSA  methods  by  Fiscal  Year  (FY)  2005, 
little  has  been  done  to  measure  the  implementation  of  PBSA  or  the  progress  towards  Dr. 
Gansler’s  goal.  In  order  to  evaluate  the  perfonnance  of  PBSA  implementation,  USAF 
decision-makers  need  to  know  the  current  state  of  PBSA  in  the  USAF.  Furthermore, 


28 


USAF  decisions  makers  need  to  know  whether  or  not  mandated  PBSA  goals  have  been 
and  will  be  met. 

This  study  addresses  the  question:  What  is  the  current  and  future  state  of  PBSA 
implementation  in  the  USAF?  Several  investigative  questions  were  developed  to  help 
answer  this  research  question.  The  five  investigative  questions  this  study  answers  are: 
IQ  1 .  Is  the  J001  database,  which  is  the  source  of  government  procurement  data, 
reliable? 

IQ2.  Is  the  USAF  meeting  interim  PBSA  goals? 

IQ3.  Is  the  USAF  on  track  to  meet  future  mandated  PBSA  goals? 

IQ4.  Is  the  percentage  of  modifications  coded  PBSA  equal  to  the  percentage  of 
non-modifications  coded  PBSA? 

IQ5.  Are  any  particular  acquisition  characteristics  associated  with  PBSA  use? 
IQ5a.  What  are  the  antecedents  to  the  use  of  PBSA? 

IQ5b.  What  are  the  barriers  to  the  use  of  PBSA? 

These  investigative  questions  were  answered  using  descriptive  statistics,  forecasting, 
contingency  tables,  and  regression. 

Research  Design 

In  order  to  properly  address  the  research  problem,  two  research  designs  were 
utilized  in  this  study;  a  time  series  design  using  archival  data  and  correlational  research. 
A  time  series  is  a  large  series  of  observations  made  on  the  same  variable  consecutively 
over  time  (Shadish,  Cook,  Campbell,  2002).  For  this  study  numerous  variables  were 
observed  over  time,  with  an  emphasis  placed  on  PBSA  contract  coding.  Just  like  many 
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other  time  series  data,  the  data  for  this  study  came  from  an  archive,  the  J001  database. 
Gaining  access  to  archival  data  for  research  use  can  sometimes  be  difficult  (Shadish, 
et.al.,  2002).  However,  by  having  a  sponsor  (AFMC/PK)  that  was  willing  to  grant 
access  to  the  needed  archival  data,  the  researcher  was  confident  that  obtaining  data  from 
the  archival  database  would  not  be  an  issue.  This  archival  data  extracted  from  the  J001 
database  for  FY  2001  through  2004  was  the  foundation  for  answering  all  five 
investigative  questions. 

Correlational  research  involves  examining  how  differences  in  one  characteristic 
or  variable  relate  to  differences  in  one  or  more  other  characteristics  or  variables  (Leedy 
and  Ormrod,  2001).  In  this  type  of  research,  surface  relationships  are  examined  without 
necessarily  probing  for  the  causal  reasons  underlying  them  (Leedy  and  Ormrod,  2001). 

A  correlation  is  deemed  to  exist  when  the  increase  or  decrease  in  one  variable,  results  in  a 
predictable  increase  or  decrease  of  another  variable  (Leedy  and  Ormrod,  2001).  Through 
the  use  of  statistical  tools,  such  as  contingency  tables  and  regressions,  this  study  searched 
for  correlations  between  PBSA  and  other  contract  variables. 

Data  Gathering 

The  majority  of  the  data  for  this  effort,  and  the  data  used  to  answer  investigative 
questions  two  through  five,  were  extracted  from  the  Air  Force  Contract  Reporting 
System,  also  know  as  the  J001.  The  J001  is  a  database  comprised  of  DD  Form  350, 
Individual  Contracting  Action  Report,  inputs  for  all  Air  Force  contracting  actions 
exceeding  $25,000.  All  DD  Form  350  data  that  are  input,  collected,  and  transmitted  is 
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considered  primary  data,  which  are  often  the  most  valid  form  of  data  (Leedy  and  Onnrod, 

2001). 

The  data  in  the  J001  database  needed  for  this  study  was  queried  using  Air  Force 
Materiel  Command’s  (AFMC)  Contracting  Business  Intelligence  System  (CBIS),  which 
receives  updates  from  the  J001.  Identifying  and  extracting  only  the  contracting  actions 
that  were  services-related  was  done  by  isolating  block  B 12A,  Federal  Stock  Class  (FSC) 
or  Service  (SVC)  Code,  on  the  DD  Fonn  350.  FY  2001  was  the  first  year  the  form  DD 
350  contained  a  block  for  coding  of  PBSA.  On  the  DD  Form  350,  block  B12A  is  a  four 
position  alphanumeric  code.  All  FSC  codes  contain  only  numeric  designations,  while  all 
SVC  codes  start  with  an  alpha  designation.  In  order  to  extract  only  the  SVC  coded 
contracting  actions,  a  search  was  conducted  for  all  actions  where  B12A  was  between 
A000  and  Z999.  The  resulting  contracting  actions,  all  of  which  were  services-related, 
were  extracted  from  the  database  and  transferred  to  an  excel  document.  While  highly 
unlikely,  it  is  possible  some  PBSA  coded  contracts  used  FSC  codes  by  mistake. 

However,  because  of  the  low  probability  of  occurrence,  no  attempt  to  extract  these 
contracts  was  made. 

Due  to  the  size  and  complexity  of  the  information  contained  on  each  DD  Form 
350,  not  all  blocks  on  the  form  were  extracted  for  analysis.  Table  2  shows  the  variables 
extracted  form  the  database. 
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Table  2.  DD  Form  350  Variables  Extracted  from  the  J001  Database. 


B 1 A  -  Contract  Number 

B1C  -  Bundled  Contract 

BID  -  Bundled  Contract  Exception 

B1E  -  Performance-Based  Service  Contract 

B3  -  Action  Date 

B4  -  Completion  Date 

B6A  -  City  or  Place  Code 

B8  -  Obligated  or  Deobligated  Dollars 

BIO  -  Multi-Year  Contract 

B1 1  -  Total  Estimated  Contract  Value 

B12A-FSC  or  SVC  Code 

B12D-NAICS  Code 

B13A  -  Contract/Order 

B13B  -  Type  of  Indefinite-Delivery  Contract 

B 13E  -  Multiple  Award  Contract  Fair 

Opportunity 


B 13F  -  Indefinite-Delivery  Contract  Use 
C 1  -  Synopsis 

C2  -  Reason  Not  Synopsized 

C3  -  Extent  Competed 

C5  -  Type  of  Contract 

C8  -  Solicitation  Procedures 

C9  -  Authority  for  Other  Than  Full  & 

Open  Competition 

C14  -  Commercial  Items 

D1A  -  Type  of  Entity 

DIB  -  Women-Owned  Business 

D1C  -  HUBZone  Representation 

DID  -  Ethnic  Group 

D4A  -  Type  of  Set-Aside 

D4B  -  Type  of  Preference 

D7  -  SBIR  Program 


The  researcher  first  reviewed  all  variables  for  applicability  to  this  study.  Variables  were 
eliminated  based  not  only  on  their  pertinence  to  PBSA,  but  also  their  frequency  of  use. 
For  example,  Report  Number  and  Recovered  Material  Clauses,  were  eliminated  from 
consideration  because  both  are  seldom  used  and  have  no  relationship  to  PBSA.  Next, 
variables  that  are  typically  included  in  contracting  metrics,  such  as  type  of  service,  dollar 
amount,  business  size,  and  contract  type,  were  selected  for  extraction.  Then  I  included 
socio-economic  variables,  such  as  Women-Owned  Business,  HUBZone  Representation, 
and  Ethnic  Group.  While  these  socio-economic  variables  are  most  likely  not  correlated 
with  the  use  of  PBSA,  they  were  selected  in  order  to  check  for  any  unusual  or  compelling 
relationships.  Curiosity  was  the  driver  for  extraction  of  a  few  variables.  On  the  surface 
theses  variables  were  expected  to  have  no  correlation  to  PBSA.  However,  no 
determination  can  be  made  until  they  are  tested.  These  variables  include  Multiple  Award 
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Fair  Opportunity,  Bundled  Contract,  and  Bundled  Contract  Exception.  Before  finalizing 
the  selection  of  variables,  a  member  of  the  research  committee  reviewed  and  agreed  that 
the  variables  identified  by  the  researcher  for  extraction  from  the  J001  database  were 
appropriate. 

Variable  Descriptions 

This  section  provides  a  description  of  possible  responses  and  expected 
relationships  to  PBSA  for  each  variable  extracted  from  the  J001  database.  This 
information  was  obtained  from  the  Contract  Action  Reporting  System  FY04  Training  and 
Desk  Guide  (Office  of  Procurement  Management,  2003).  Because  socio-economic 
variables,  such  as  Type  of  Entity,  Women-Owned  Business,  HUBZone  Representation, 
Ethnic  Group,  Type  of  Set-Aside,  Type  of  Preference,  and  SBIR  Program,  have  no 
practical  correlation  to  PBSA,  no  expected  relationships  were  noted. 

Contract  Number.  This  is  a  13-character  alphanumeric  designation  procurement 
identification  number.  This  variable  is  used  for  informational  purposes  only  and  was  not 
tested  for  relationships  with  PBSA. 

Bundled  Contract.  This  variable  is  coded  “Y”  when  the  contract  meets  the 
definition  of  “bundled  contract”  and  the  contract  value  exceeds  $5  million.  This  variable 
is  coded  “N”  when  code  “Y”  dose  not  apply.  This  variable  was  expected  to  have  a 
negative  correlation  with  PBSA  use  when  coded  “Y”  because  in  order  for  a  bundled 
contract  to  be  performance-based,  a  majority  of  the  consolidated  contracts  would  have 
also  had  to  be  performance-based. 
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Bundled  Contract  Exception.  If  Bundled  Contract  is  coded  “N”,  this  variable  is 

left  blank;  otherwise  one  of  the  following  codes  is  entered: 

Code  A  -  The  procurement  is  mission  critical  and  the  agency  has  detennined  that 
the  consolidation  of  requirements  is  critical  to  the  agency’s  mission. 

Code  B  -  The  agency  used  the  OMB  Circular  A-76  process  to  determine  that  the 
consolidation  of  requirements  is  necessary  and  justified. 

Code  C  -  Codes  A  and  B  do  not  apply 

This  variable  was  expected  to  have  a  negative  correlation  with  the  probability  of  PBS  A 
because  it’s  dependent  on  the  coding  of  “Y”  for  Bundled  Contract. 

Action  Date.  The  year,  month,  and  day  of  fiscal  obligation.  Although  descriptive 
in  nature,  it  was  expected  that  this  variable,  over  time,  would  have  a  positive  correlation 
with  PBSA.  This  was  expected  because  it  was  assumed  that  PBSA  percentages  were 
going  to  increase  over  time. 

Completion  Date.  The  year,  month,  and  day  of  the  last  contract  delivery  or  the 
end  of  the  perfonnance  period.  Although  descriptive  in  nature,  it  was  also  expected  that 
this  variable,  over  time,  would  have  a  positive  correlation  with  PBSA.  This  was  expected 
because  it  was  assumed  that  PBSA  percentages  were  going  to  increase  over  time. 

Obligated  or  Deobligated  Dollars.  This  variable  is  the  net  amount  of  funds 
obligated  or  deobligated  by  the  contracting  action.  This  variable  was  expected  to  have  a 
positive  correlation  with  PBSA  as  the  value  increased.  This  was  expected  because  the 
complex  nature  of  PBSA  contracts  implies  that  the  time  and  effort  needed  to  establish  a 
PBSA  contract  would  not  be  done  for  smaller  dollar  contracts. 

Mult-Year  Contract.  This  variable  is  coded  “Y”  when  the  contracting  action  is  a 
multi-year  contract  and  coded  “N”  when  code  “Y”  does  not  apply.  It  was  expected  that 
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this  variable  would  have  no  correlation  with  PBSA  because  most  multi-year  contracts  are 
avoided  when  at  all  possible. 

Total  Estimated  Contract  Value.  This  variable  is  the  net  amount  of  the  total 
estimated  contract  value  at  the  time  of  the  initial  contract.  This  includes  placement  of  an 
indefinite-delivery  or  multi-year  contract.  Additionally,  this  variable  includes  the  total 
estimated  value  of  orders  and  option  anticipated  over  the  life  of  the  contract.  This 
variable  was  expected  to  have  a  positive  correlation  with  PBSA  for  the  same  reasons 
indentified  for  Obligated  or  Deobligated  Dollars. 

FSC  or  SVC  Code.  This  variable  is  the  4-characrter  code  that  describes  the 
contract  effort.  Each  effort  falls  into  one  of  three  categories;  Supplies,  Services,  or 
Research,  Development,  Test  and  Evaluation.  This  variable  was  expected  to  have  a 
positive  correlation  with  PBSA  when  general  base  services,  such  as  grounds  maintenance 
or  custodial  services,  were  selected.  R&D  services  were  expected  to  have  a  negative 
correlation  with  PBSA  because  of  their  seldom  use  of  perfonnance-based  contracts  in  the 
past. 

NAICS  Code.  This  variable  stands  for  the  North  American  Industry  Classification 
System  (NAICS)  and  also  helps  identify  the  product  of  service.  It  is  important  to  note 
that  more  than  one  code  may  apply.  This  variable  was  expected  to  act  similar  to  FSC  or 
SVC  Code  because  it  is  a  function  of  the  FSC  or  SVC  Code. 

Contract/Order.  This  variable  is  coded  using  one  of  the  following  eight  codes: 

Code  1  -  Letter  Contract.  Code  1  is  entered  when  the  contracting  action  is  a  letter 
contract  or  a  modification  to  a  letter  contract  that  has  not  been  definitized. 

Code  3  -  Definitive  Contract.  Code  3  is  entered  when  the  contracting  action  is  the 
award  or  modification  of  a  definitive  contract  or  a  modification  that  definitizes  a 
contract. 
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Code  4  -  Order  under  an  Agreement.  Code  4  is  entered  when  the  contracting 
action  is  an  order  or  definitization  of  an  order  under  an  agreement  other  than  a 
blanket  purchase  agreement. 

Code  5  -  Order  under  Indefinite-Delivery  Contract.  Code  5  is  entered  when  the 
contracting  action  is  an  order,  including  a  task  or  delivery  order,  under  an 
indefinite-delivery  contract  awarded  by  a  Federal  agency,  such  as  a  GSA 
indefinite-delivery  contract. 

Code  6  -  Order  under  Federal  Schedule.  Code  6  is  entered  when  the  contracting 
action  is  an  order  under  a  GSA  or  VA  Federal  Supply  Schedule,  or  a  call  against  a 
blanket  purchase  agreement  established  under  a  GSA  or  VA  Federal  Supply 
Schedule. 

Code  7  -  Blanket  Purchase  Agreement  Order  under  Federal  Schedule.  Code  7  is 
entered  when  the  contracting  action  is  a  BPA  order  under  a  GSA  or  VA  Federal 
Supply  Schedule. 

Code  8  -  Order  from  Procurement  List.  Code  8  is  entered  when  the  contracting 
action  is  an  action  placed  with  Federal  Prison  Industries  (UNICOR)  or  a  JWOD 
Participating  Nonprofit  Agency. 

Code  9  -  Award  under  FAR  Part  13.  Code  9  is  entered  when  the  contracting 
action,  including  an  action  in  a  designated  industry  group  under  the  Small 
Business  Competitiveness  Demonstration  Program,  is  an  award  using  simplified 
acquisition  procedures. 


This  variable  was  expected  to  have  negative  correlation  with  PBSA  when  coded  “4”,  “5”, 
“6”,  “7”,  or  “8.”  This  was  expected  because  PBSA  contracts  are  rarely  used  when 
placing  an  order  off  of  an  existing  contract. 

Type  of  Indefinite-Delivery  Contract.  This  variable  is  coded  using  one  of  the 
following  three  codes  when  Contract/Order  is  coded  “Order”  for  the  variable  Indefinite- 
Delivery  Contract: 

Code  A  -  The  action  pertains  to  a  requirements  contract. 

Code  B  -  The  action  pertains  to  an  indefinite-quantity  contract. 

Code  C  -  The  action  pertains  to  a  definite-quantity  contract. 

Because  indefinite  delivery  contracts  are  seldom  PBSA,  it  was  expected  that  this  variable 

would  have  a  negative  correlation  with  PBSA  use. 

Multiple  Award  Contract  Fair  Opportunity.  This  variable  is  coded  using  one  of 

the  following  five  codes: 
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Code  A  -  Fair  Opportunity  Process.  Code  A  is  entered  when  the  delivery  or  task 
order  was  issued  pursuant  to  a  process  that  pennitted  each  contract  awardee  a  fair 
opportunity  to  be  considered. 

Code  B  -  Urgency.  Code  B  is  entered  when  the  agency  need  is  so  urgent  that 
providing  a  fair  opportunity  would  result  in  unacceptable  delays. 

Code  C  -  One/Unique  Source.  Code  C  is  entered  when  only  one  contract  awardee 
is  capable  of  providing  the  supplies  or  services  at  the  level  or  quality  required 
because  the  supplies  or  services  are  unique  or  highly  specialized. 

Code  D  -  Follow-On  Contract.  Code  D  is  entered  when  the  order  was  issued  on  a 
sole-source  basis  in  the  interest  of  economy  and  efficiency  as  a  logical  follow-on 
to  an  order  already  issued  under  the  contract. 

Code  E  -  Minimum  Guarantee.  Code  E  is  entered  when  it  was  necessary  to  place 
an  order  to  satisfy  a  minimum  amount  guaranteed  to  the  contractor. 

This  variable  was  expected  to  have  a  positive  correlation  with  PBSA  when  “A”  was 

coded  because,  according  to  the  literature,  PBSA  increases  competition,  thus  enhancing 

fair  opportunity  processes. 

Indefinite-Delivery  Contract  Use.  This  variable  is  coded  using  one  of  the  four 

following  codes  if  Type  of  Indefinite -Delivery  Contract  was  coded  and  the  action  is  the 

initial  placement  of  an  indefinite-delivery  contract: 

Code  A  -  The  contract  can  be  used  Government- wide. 

Code  B  -  The  contract  can  be  used  within  the  DoD  only. 

Code  C  -  The  contract  can  be  used  within  the  department  only. 

Code  D  -  The  contract  can  be  used  by  the  contracting  office  only. 

Because  indefinite  delivery  contracts  are  seldom  PBSA,  it  was  expected  that  this  variable 

would  have  a  negative  correlation  with  PBSA  use. 

Synopsis.  This  variable  is  coded  using  one  of  the  three  following  codes: 

Code  A  -  Only  a  synopsis  of  the  proposed  action  was  prepared. 

Code  B  -  A  combined  synopsis/solicitation  of  the  proposed  action  was  prepared. 
Code  N  -  A  synopsis  was  not  prepared. 
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This  variable  was  expected  to  have  a  negative  correlation  with  PBSA  when  coded  “N” 
because  PBSA  is  best  used  in  competitive  circumstances.  Not  preparing  a  synopsis  limits 
the  notification  of  upcoming  solicitations,  thus  limiting  competition. 

Reason  Not  Synopsized.  This  variable  is  left  blank  unless  Synopsis  is  coded  “N.” 
If  Synopsis  is  coded  “N”  then  this  variable  is  coded  using  one  of  the  three  following 
codes: 


Code  A  -  The  action  was  not  synopsized  due  to  urgency. 

Code  B  -  The  action  was  not  synopsized  because  the  acquisition  was  made 
through  another  means  that  provided  access  to  the  notice  of  the  proposed  action 
through  a  single,  Government-wide  point  of  entry. 

Code  Z  -  The  action  was  not  synopsized  due  to  some  other  reason. 

This  variable  was  expected  to  have  a  negative  correlation  with  PBSA  because  not 

synopsizing  results  in  limited  competition. 

Extent  Competed.  This  variable  is  coded  using  one  of  the  four  following  codes: 

Code  A  -  The  action  was  competed. 

Code  B  -  The  action  is  not  available  for  competition. 

Code  C  -  The  action  is  a  follow-on  to  competed  action. 

Code  D  -  The  action  was  not  competed. 

Because  PBSA  is  best  used  in  competitive  circumstance,  codes  “A”  and  “C”  for  this 

variable  were  expected  to  have  a  positive  correlation  with  PBSA  and  codes  “B”  and  “D” 

were  expected  to  have  a  negative  correlation. 

Type  of  Contract.  This  variable  identifies  the  type  of  contract  utilized  for  the 

action  and  is  coded  using  one  of  the  12  following  codes: 

Code  A  -  Fixed-Price  Redetermination. 

Code  J  -  Firm-Fixed-Price. 

Code  K  -  Fixed-Price  Economic  Price  Adjustment. 

Code  L  -  Fixed-Price  Incentive. 

Code  M  -  Fixed-Price- Award-Fee. 

Code  R  -  Cost-Plus-Award-Fee. 
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Code  S  -  Cost  Contract. 

Code  T  -  Cost-Sharing. 

Code  U  -  Cost-Plus-Fixed-Fee. 

Code  V  -  Cost-Plus-Incentive-Fee. 

Code  Y  -  Time-and-Materials. 

Code  Z  -  Labor-Hour. 

It  was  expected  that  this  variable  would  have  a  positive  correlation  with  PBSA  when 

coded  “J”  because  firm-fixed  price  contracts  have  been  identified  in  the  literature  as  the 

ideal  contract  types  for  PBSA.  Conversely,  all  the  cost-plus  contract  types  were  expected 

to  have  a  negative  correlation  with  PBSA  because  the  use  of  these  types  of  contracts  is 

discouraged  and  not  conducive  to  PBSA  methods. 

Solicitation  Procedures.  This  variable  is  left  blank  if  the  action  is  pursuant 

simplified  acquisition  procedures  or  is  an  order  or  call  under  a  Federal  schedule. 

Otherwise,  the  variable  is  coded  using  one  of  the  following  nine  codes: 

Code  A  -  Full  and  Open  Competition  Sealed  Bid. 

Code  B  -  Full  and  Open  Competition— Competitive  Proposal. 

Code  C  -  Full  and  Open  Competition— Combination. 

Code  D  -  Architect-Engineer. 

Code  E  -  Basic  Research. 

Code  F  -  Multiple  Award  Schedule. 

Code  G  -  Alternative  Sources. 

Code  K  -  Set-Aside. 

Code  N  -  Other  than  Full  and  Open  Competition. 

Because  PBSA  is  best  used  in  competitive  circumstance,  codes  “B”  and  “C”  for  this 
variable  were  expected  to  have  a  positive  correlation  with  PBSA  and  code  “N”  was 
expected  to  have  a  negative  correlation.  Additionally,  PBSA  is  not  appropriate  for  sealed 
bidding  and  is  not  required  for  architect-engineering  services;  therefore  codes  “A”  and 
“D”  were  also  expected  to  have  a  negative  correlation  with  PBSA. 
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Authority  for  Other  Than  Full  and  Open  Competition.  This  variable  is  left  blank 
unless  Solicitation  Procedures  is  coded  “N.”  If  Solicitation  Procedures  is  coded  “N”  then 
one  of  the  following  14  codes  is  used: 

Code  1A  -  Unique  Source. 

Code  IB  -  Follow-On  Contract. 

Code  1C  -  Unsolicited  Research  Proposal. 

Code  ID  -  Patent  or  Data  Rights. 

Code  IE  -  Utilities. 

Code  IF  -  Standardization. 

Code  1G  -  Only  One  Source— Other. 

Code  2A  -  Urgency. 

Code  3A  -  Particular  Sources. 

Code  4A  -  International  Agreement. 

Code  5A  -  Authorized  by  Statute. 

Code  5B  -  Authorized  Resale. 

Code  6A  -  National  Security. 

Code  7A  -  Public  Interest. 

Because  PBSA  is  best  used  in  competitive  circumstance  this  variable  was  expected  to 
have  a  negative  correlation  with  PBSA. 

Commercial  Items.  This  variable  is  coded  “Y”  for  yes  or  “N”  for  no  depending 
on  the  inclusion  of  FAR  clause  52.212-4,  Contract  Tenns  and  Conditions — Commercial 
Items,  in  the  contract.  If  this  clause  is  in  the  contract  then  the  action  was  awarded  as  a 
commercial  item  and  thus  is  code  “Y.”  Because  commercial  items  cover  such  a  broad 
range  of  services,  the  researcher  was  unsure  what  effect,  if  any,  this  variable  would  have 
on  PBSA. 

Type  of  Entity.  This  variable  describes  the  type  of  business  entity  the  action  was 
awarded  to  and  is  coded  using  one  of  the  following  1 1  codes: 

Code  A  -  Small  Disadvantaged  Business  (SDB)  Perfonning  in  U.S. 

Code  B  -  Other  Small  Business  (SB)  Perfonning  in  U.S. 

Code  C  -  Large  Business  Performing  in  U.S. 

Code  D  -  JWOD  Participating  Nonprofit  Agency. 
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Code  F  -  Hospital. 

Code  L  -  Foreign  Concern  or  Entity. 

Code  M  -  Domestic  Firm  Performing  Outside  U.S. 

Code  T  -  Historically  Black  College  or  University  (HBCU). 

Code  U  -  Minority  Institution  (MI). 

Code  V  -  Other  Educational. 

Code  Z  -  Other  Nonprofit. 

Women-Owned  Business.  This  variable  identifies  whether  or  not  the  contractor 
identifies  themselves  as  a  women-owned  business  and  is  coded  “Y”  for  yes,  “N”  for  no, 
or  “U”  for  uncertified. 

HUBZone  Representation.  This  variable  identifies  whether  or  not  the  contractor 
represented  that  it  is  a  HUBZone  small  business  concern  and  is  coded  “Y”  for  yes  and 
“N”  for  no. 

Ethnic  Group.  This  variable  identifies  the  ethnic  group  for  the  contractor,  if 

applicable.  This  variable  is  left  blank  unless  the  action  is  with  a  small  disadvantaged 

business.  Otherwise  one  of  the  following  seven  codes  is  used: 

Code  A  -  Asian-Indian  American. 

Code  B  -  Asian-Pacific  American. 

Code  C  -  Black  American. 

Code  D  -  Hispanic  American. 

Code  E  -  Native  American. 

Code  F  -  Other  SDB  Certified  or  Determined  by  SBA. 

Code  Z  -  No  Representation. 

Type  of  Set-Aside.  This  variable  identifies  the  type  of  set-aside  used  and  is  coded 

using  one  of  the  following  1 1  codes: 

Code  A  -  None. 

Code  B  -  Total  SB  Set-Aside. 

Code  C  -  Partial  SB  Set-Aside. 

Code  D  -  Section  8(a)  Set-Aside  or  Sole  Source. 

Code  E  -  Total  SDB  Set-Aside. 

Code  F  -  HBCU  or  MI  -  Total  Set-Aside. 

Code  G  -  HBCU  or  MI  -  Partial  Set-Aside. 
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Code  H  -  Very  Small  Business  Set-Aside. 

Code  J  -  Emerging  Small  Business  Set-Aside. 

Code  K  -  HUBZone  Set-Aside  or  Sole  Source. 

Code  L  -  Combination  HUBZone  and  8(a). 

Type  of  Preference.  This  variable  identifies  the  type  pf  preference  given  to  small 
businesses  and  is  coded  using  one  of  the  following  five  codes: 

Code  A  -  None. 

Code  B  -  SDB  Price  Evaluation  Adjustment— Unrestricted. 

Code  C  -  SDB  Preferential  Consideration— Partial  SB  Set-Aside. 

Code  D  -  HUBZone  Price  Evaluation  Preference. 

Code  E  -  HUBZone  Price  Evaluation  Preference  and  SDB  Price  Evaluation 
Adjustment. 

SBIR  Program.  This  variable  identifies  whether  or  not  the  action  is  related  to  the 

Small  Business  Innovation  Research  (SBIR)  Program  and  if  so,  what  phase  of  it.  This 

variable  is  coded  using  the  following  four  codes: 

Code  A  -  Not  a  SBIR  Program  Phase  I,  II,  or  III. 

Code  B  -  SBIR  Program  Phase  I  Action. 

Code  C  -  SBIR  Program  Phase  II  Action. 

Code  D  -  SBIR  Program  Phase  III  Action. 


Data  Reliability 

Despite  its  wide  use,  the  J001  and  other  federal  procurement  databases  have  been 
proven  to  have  their  flaws.  A  2003  report  from  the  General  Accounting  Office  (GAO) 
noted,  “FPDS  data  are  inaccurate  and  incomplete  (Woods,  2003).”  For  instance,  in  the 
2001  review  the  GAO  found  that  the  value  of  contracts  awarded  to  HUBZone  firms  could 
have  been  hundreds  of  millions  of  dollars  different  than  reported  (Woods,  2003).  They 
also  found  instances  where  multiple  orders  were  reported  as  a  single  transaction.  For 
instance,  an  order  reported  as  $1 1,443,000  should  have  been  reported  as  87  separate 
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actions  at  or  below  $25,000  (Woods,  2003).  Because  of  these  inaccuracies,  the  GAO  has 
been  unable  to  assess  the  implementation  of  procurement  programs  and  limited  its 
reliance  on  FPDS  data.  The  GAO  found  similar  inaccuracies  with  the  Department  of 
Defense’s  data  system,  which  feeds  information  into  FPDS.  However,  in  the  report  no 
details  were  given  on  the  specifics  of  the  DoD’s  inaccuracies. 

Of  the  problems  noted  in  the  GAO  findings,  none  specifically  pertained  to  the  use 
or  coding  of  PBSA.  The  only  remark  suggesting  possible  problems  with  proper  coding 
of  PBS  A  actions  included,  “In  some  cases,  those  processing  the  data  did  not  have  a 
complete  understanding  of  the  infonnation  requirements.”  This  remark  suggests  that 
some  contracting  actions  may  or  may  not  be  being  coded  PBSA  accurately.  Another 
GAO  report  further  supports  the  possibility  of  inappropriate  coding  of  federal 
procurement  data  by  noting  there  are  inconsistencies  in  the  interpretation  of  the  definition 
of  performance  based  contracts  (Woods,  2002).  In  addition,  personal  experiences  by  the 
researcher  suggest  that  time  constraints,  pressure  to  meet  procurement  goals,  and  a 
general  lack  of  emphasis  on  proper  coding  may  also  attribute  to  inaccuracies  in  the 
databases. 

Builders  of  the  input  system  for  the  DD  Form  350  realized  that  there  was  a 
possibility  for  user  error  when  inputting  procurement  information.  In  order  to  counter 
this,  the  system  has  built-in  error  checking  capability.  For  instance,  if  block  B  1C, 
Bundled  Contract,  is  coded  “N”  for  no,  the  system  will  not  allow  you  to  code  an 
exception  in  block  BID,  Bundled  Contract  Exception.  Although  this  coding  check 
capability  does  not  account  for  subjective  coding  decisions,  it  does  add  to  the  overall 
reliability  of  the  data. 
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Data  Analysis 

AFI  63-124,  the  guiding  document  for  USAF  PBSA,  states  that  all  services 
contracts  over  the  Simplified  Acquisition  Threshold  (SAT),  which  has  been  listed  at 
$100,000  during  the  years  pertaining  to  this  study,  are  eligible  for  PBSA,  excluding  the 
services  listed  in  FAR  Subpart  37. 102  (DAF,  2004).  The  services  excluded  from  PBSA 
use  include  architect-engineering  services,  construction,  utility  services,  and  services  that 
are  identical  to  supply  purchases  (FAR  Part  37).  Because  these  services  are  not  eligible 
for  PBSA  and  none  of  them  were  coded  PBSA,  they  were  removed  from  the  J001  data  by 
identifying  and  deleting  the  corresponding  service  code  categories.  All  services,  with  the 
exception  of  R&D  type  services,  and  construction  work  are  broken  up  into  the  23  major 
categories  listed  in  Appendix  A.  Each  category  has  been  assigned  a  four  position  code, 
also  known  as  SVC  code,  starting  with  a  letter  followed  by  three  numbers.  The  letter 
identifies  the  major  category  and  the  numbers  identify  the  specific  service  within  each 
category.  All  contracts  that  have  a  SVC  code  that  begins  with  the  alpha  designation  “C” 
are  architect  and  engineering  service  contracts.  All  contracts  that  have  a  SVC  code  that 
begins  with  the  alphanumeric  designations  “SI”  are  utility  service  contracts.  All 
contracts  that  have  a  SVC  code  that  begins  with  the  alpha  designation  “Y”  are 
construction  contracts.  Accordingly,  all  actions  where  block  B12A,  FSC  or  SVC,  began 
with  C,  SI,  or  Y  were  removed.  After  removing  all  exempt  services,  the  remaining 
contracting  actions,  all  of  which  were  greater  than  $100K  and  not  excluded  under  FAR 
Subpart  37.102,  were  analyzed. 
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The  data  were  analyzed  using  descriptive  statistics,  forecasting,  regression,  and 
contingency  tables.  The  following  sections  describe  how  these  methods  were  used  to 
answer  the  five  investigative  questions. 

Investigative  Question  1.  In  order  to  determine  the  reliability  of  the  J001,  and 
answer  investigative  question  1,  a  stratified  random  sample  of  PBS  A  contract  files  were 
obtained  from  three  different  contracting  offices  located  at  Wright-Patterson  Air  Force 
Base  (WPAFB).  These  offices  included  88  ABW/PK,  AFRL  Det  1/PK,  and  ASC/PK. 
These  three  offices  were  selected  because  they  each  perform  a  different  contracting 
function  for  the  USAF.  The  88th  ABW/PK  is  an  operational  contracting  squadron  that 
primarily  supports  base  operations;  ASC/PK  supports  a  variety  of  weapon  systems 
related  requirements,  and  AFRL  Det  1/PK  contracts  primarily  for  research  and 
development  (R&D)  efforts.  Together  these  offices  handle  a  wide  variety  of  contracts, 
from  standard  firm  fixed  price  base  custodial  contracts  to  complex  cost  reimbursable 
R&D  contracts  to  major  weapon-system  related  contracts. 

The  goal  was  to  evaluate  at  least  30  randomly  selected  contracts  from  each  office. 
In  order  to  ensure  an  appropriate  amount  of  contracts  were  selected  from  each  office,  the 
contracts  were  first  stratified.  Using  the  data  from  the  J001,  all  PBSA  coded  contract 
data  from  each  office  was  identified  using  each  office’s  unique  Department  of  Defense 
Activity  Address  Code  (DoDAAC)  and  office  code.  The  DoDAAC  is  an  alpha  numeric 
designation  that  is  always  the  first  six  designations  of  a  contract  number  and  is  usually 
the  same  for  all  contracts  in  a  particular  office.  The  office  code  is  a  unique  alpha  numeric 
designation  identifier  that  helps  track  actions  performed  by  a  particular  office  and  is 
required  when  completing  a  form  DD  350. 
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Once  the  contract  data  for  each  office  was  identified,  extracted,  and  sorted  by  date 
of  action,  a  random  sample  for  each  office  was  generated  using  a  random  number 
generator  software  program  tool.  Those  contracts  randomly  selected  were  requested  for 
review.  In  some  instances  the  contracting  offices  were  unable  to  locate  a  contract  file 
because  they  were  a)  already  staged  away  in  storage  at  a  different  location,  b)  currently 
being  used,  c)  classified,  or  d)  simply  unable  to  be  located.  In  these  instances,  a 
replacement  contract  from  the  randomized  list  was  identified  and  substituted. 

The  stratified  sample  of  service  contracts  coded  PBSA  in  the  J001  was  then 
evaluated  against  the  four  PBSA  contract  criteria  identified  in  The  Guidebook  for 
Performance-Based  Services  Acquisitions  in  the  Department  of  Defense  (PBSA 
Guidebook).  The  criteria  identified  in  the  PBSA  Guidebook  were  selected  in  lieu  of  the 
four  PBSA  criteria  identified  in  the  Federal  Acquisition  Regulation  (FAR)  because, 
unlike  the  FAR,  the  PBSA  Guidebook  criteria  is  a  minimum  mandatory  requirement  and 
was  published  by  Dr.  Gansler,  the  same  person  that  established  the  FY  2005  goal  that 
50%  of  all  services  be  performance  based.  According  to  the  PBSA  Guidebook,  in  order 
for  an  acquisition  to  be  considered  performance-based,  it  must  meet  the  following 
minimum  requirements: 

(1)  Perfonnance  Work  Statement  -  The  perfonnance  work  statement  describes 
the  requirement  in  terms  of  measurable  outcomes  rather  than  by  means  of 
prescriptive  methods. 

(2)  Measurable  Perfonnance  Standards  -  To  determine  whether  performance 
outcomes  have  been  met,  measurable  performance  standards  define  what  is 
considered  acceptable  perfonnance. 

(3)  Remedies  -  Remedies  are  procedures  that  address  how  to  manage 
performance  that  does  not  meet  performance  standards.  While  not  mandatory, 
incentives  should  be  used,  where  appropriate,  to  encourage  perfonnance  that 
will  exceed  perfonnance  standards.  Remedies  and  incentives  complement 
each  other. 
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(4)  Performance  Assessment  Plan  -  This  plan  describes  how  contractor 

performance  will  be  measured  and  assessed  against  performance  standards. 
(Quality  Assurance  Plan  or  Quality  Surveillance  Plan). 

All  evaluations  were  conducted  by  the  researcher,  who  is  Level  II  Certified  in 

Contracting  through  the  Acquisition  Professional  Development  Program,  and  considered 

a  subject  matter  expert  in  PBSA.  For  the  purposes  of  this  study,  contracts  were  evaluated 

on  the  existence  of  the  four  PBSA  factors  mentioned  above.  A  contract  was  determined 

to  be  correctly  coded  PBSA  if  it  contained  all  four  PBSA  criteria,  even  if  not  named 

exactly  as  stated  in  the  PBSA  Guidebook.  For  instance,  a  Statement  of  Work  (SOW)  that 

described  the  requirement  in  terms  of  measurable  outcomes  was  considered  to  meet  the 

first  criteria  of  the  PBSA  Guidebook,  even  though  it  was  not  named  a  Performance  Work 

Statement  (PWS).  A  contract  was  determined  to  be  incorrectly  coded  PBSA  if  it  did  not 

meet  one  or  more  of  the  four  PBSA  criteria.  Because  determining  when  performance 

incentives  are  appropriate  is  so  subjective  and  the  use  of  incentives  is  so  infrequent,  the 

third  PBSA  criterion,  “Remedies,”  was  assumed  to  have  been  correctly  applied  for  all 

PBSA  contracts. 

Additionally,  all  PBSA  coded  contracts  from  AFRL  were  evaluated,  not  using  the 
criteria  identified  in  the  PBSA  Guidebook,  but  using  the  detenninations  made  in  AFRL’s 
Management  and  Oversight  of  Acquisition  of  Services  Processes  (MOASP).  Prior  to 
May  7,  2004,  all  Research  and  Development  (R&D)  contracts  had  followed  the 
requirements  for  supplies  (AFRL,  2004).  However,  because  AFRL’s  R&D  requirements 
are  inherently  performance-based,  they  drafted  their  own  MOASP  describing  how  they 
meet  the  principles  of  AFI  63-124.  AFRL’s  MOASP  was  approved  by  the  Air  Force 
Program  Executive  Officer  for  Combat  &  Mission  Support  (AFPEO/CM)  and  authority 
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to  act  as  the  Designation  Official  for  the  management  and  oversight  of  all  services 
acquisitions  within  AFRL  was  delegated  to  the  AFRL  commander.  Therefore,  as  of  May 
7  2004,  all  AFRL  R&D  contracts  are  considered  services  and  coded  PBSA  in  accordance 
with  the  approved  AFRL  MOASP.  (DAF,  AFRL  MOASP,  2003) 

Under  the  AFRL  MOASP,  R&D  contracts  meet  the  principles  of  AFI  63-124 
because: 

1)  Contractor  written  SOWs  in  response  to  Government  issued  objectives, 
describe  work  in  terms  of  what  the  required  output  is  rather  than  how  the  work 
is  to  be  performed. 

2)  Oversight  between  the  objective  and  outcome  is  accomplished  through  the 
selection  of  appropriate  reporting  requirements,  such  as  Contract  Data 
Requirements  Lists  (CDRLs),  Program  Management  Reviews  (PMRs), 
Technical  Management  Reviews  (TMRs),  electronic  management  information 
systems,  and  interaction  between  the  contractor  and  the  government. 

3)  Deliverables  may  be  final  reports,  prototypes,  other  hardware  or  software,  etc. 


The  sample  of  AFRL  contract  files  were  evaluated  against  these  criteria  in  the  same 
manner  the  other  offices’  contracts  were  evaluated. 

Lastly,  all  PBSA  service  contracts  for  Contractor  Engineering  and  Technical 
Services  (GETS)  covered  by  AFI  21-110,  Engineering  and  Technical  Sendees,  were 
evaluated  in  accordance  with  Supplement  1  to  AFI  63-124,  which  states  that  the 
surveillance  requirements  of  AFI  63-124  do  not  apply  (DAF,  Sup  1,  1999).  AFMC 
MOASP  dated  February  2004  rescinded  this  exception.  Therefore,  all  CETS  contracts 
prior  to  February  2004  were  evaluated  on  only  one  criteria;  the  existence  of  a 
performance  work  statement  that  described  the  requirement  in  terms  of  measurable 
outcomes  rather  than  by  means  of  prescriptive  methods. 
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Once  the  contract  evaluations  were  complete,  a  large  sample,  one-tailed  test  about 
a  population  proportion  was  used  to  statistically  test  the  reliability  of  the  sample  and 
answer  the  question  of  whether  PBSA  coded  contracts  are  coded  consistent  with  the 
minimum  mandatory  criteria  identified  in  the  PBSA  Guidebook  (McClave,  et  al.,  2001). 
Ideally  all  PBSA  contracts  would  be  coded  correctly.  However,  like  any  other  process 
there  are  bound  to  be  mistakes,  or  defects,  in  coding  of  contracts.  Because  of  this,  an 
acceptable  threshold  was  established.  The  researcher  felt  that  PBSA  coding  could  be 
considered  reliable  if  less  than  10%  of  the  PBSA  contracts  were  coded  incorrectly. 

Inferences  about  population  proportion,  or  percentages,  can  be  made  in  the 
context  of  the  probability,/?,  of  success  or  failure  for  a  binomial  distribution  (McClave,  et 
al.,  2001).  For  this  study  p  represented  the  percentage  of  PBSA  contracts  that  did  not 
meet  the  mandatory  minimum  criteria  identified  in  the  PBSA  Guidebook.  Additionally, 
p0  represented  the  hypothesized  percentage  of  PBSA  contracts  that  did  not  meet  the 
mandatory  minimum  criteria  identified  in  the  PBSA  Guidebook  and  was  equal  to  .10. 

The  sample  proportion  is  simply  the  sample  mean  of  the  outcomes  of  the  trials. 
According  to  the  Central  Limit  Theorem,  the  sample  mean  is  approximately  normally 
distributed  for  large  samples.  Therefore,  the  standard  normal  z  was  able  to  be  used  for 
the  test  statistic: 

Test  Statistic:  z  =  Sample  proportion  -  Null  hypothesized  proportion 

Standard  deviation  of  sample  proportion 

_  p-po 
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The  symbol  (j.  equals  —  ,  where  pq  is  estimated  by  p0q0,  q0  =  1  -  p0,  and  n  is  the  sample 
p  V  n 


size. 

However,  before  conducting  any  type  of  hypothesis  test,  the  researcher  had  to 
check  to  detennine  whether  the  sample  size  was  large  enough  to  use  the  nonnal 
approximation  for  the  sampling  distribution  of  the  sample  proportion,  p  (McClave,  et  al., 
2001).  In  order  to  use  the  large  sample  test  of  hypothesis  about  a  population  proportion, 
the  sample  size  must  be  large  enough  that  interval pQ±  3  <j  does  not  include  0  or  1 .  This 

ensures  that  the  sample  size  is  large  enough  to  assume  that  the  nonnal  distribution  will 
provide  a  reasonable  approximation  of  p  (McClave,  et  al.,  2001).  As  long  as  this  interval 
is  completely  contained  in  the  interval  0  to  1,  the  nonnal  approximation  for  the  sampling 
distribution  of  p  is  reasonable.  If  at  least  30  samples  from  each  of  the  three  offices  were 
evaluated,  the  total  number  of  samples,  n,  would  be  equal  to  90.  With  90  samples  and  a 
p0  of  .10,  interval  p0±  3  <j  equals  .10  ±  .095  or  (.005  and  .195),  both  of  which  do  not 

include  0  or  1 . 

Finally,  in  order  to  statistically  test  the  reliability  of  the  DD  Form  350  data  and 

answer  the  question  of  whether  or  not  PBS  A  coded  contracts  are  coded  consistent  with 

the  minimum  mandatory  criteria  identified  in  the  PBSA  Guidebook,  the  following  null 

and  alternate  hypotheses  were  established  (Bain  and  Englehardt,  1987): 

Null  hypothesis  #1  (H0i):  p  >  .10 
Alternate  hypothesis  #1  (Hai):  p  <  .10 

For  this  test  an  alpha  (a)  of  .01  was  used,  a  represents  the  significance  level,  or  the 
probability  of  rejecting  the  null  hypothesis  when  it  is  in  fact  true,  also  known  as  a  Type  I 


50 


error  (McClave,  et  al.,  2001).  Using  the  standard  normal  distribution,  the  appropriate 
rejection  region  for  the  specified  value  of  a  was  found.  Using  a  =  .01,  the  one-tailed 
rejection  region  is: 

Rejection  region:  z  <  -z.oi  =  -2.33 

Therefore,  the  null  hypothesis  was  rejected  if  the  value  of  the  test  statistic,  z,  was  less 
than  -2.33,  thus  falling  in  the  rejection  region. 

Investigative  Question  2.  In  order  to  answer  investigative  question  2,  the 
following  hypotheses  were  established: 

Null  hypothesis  #2  (H02):  The  USAF  is  meeting  interim  PBSA  goals. 

Alternate  hypothesis  #2  (Ha2):  The  USAF  is  not  meeting  interim  PBSA  goals. 

The  actual  percentage  of  eligible  dollars  awarded  using  PBSA  were  compared  to 
the  interim  PBSA  goals  established  by  the  acting  Under  Secretary  of  Defense  for 
Acquisition,  Technology  and  Logistics,  Mr.  Michael  Wynne.  Mr.  Wynne’s  goals  did  not 
address  the  percentage  of  actions  PBSA,  only  the  percentage  of  dollars  PBSA.  These 
interim  PBSA  goals  are  included  below. 

FY  2003  25  percent  of  dollars  awarded 

FY  2004  35  percent  of  dollars  awarded 

The  following  sub-hypotheses  were  established  in  order  to  address  the  goals  for 
each  FY  and  assist  in  detennining  whether  to  accept  or  reject  null  hypothesis  #2: 

Null  hypothesis  #2a  (Ho2a):  FY  2003  Actual  %  of  Dollars  PBSA  >  25  %. 

Alternate  hypothesis  #2a  (Ha2a):  FY  2003  Actual  %  of  Dollars  PBSA  <  25%. 

Null  hypothesis  #2b  (H02b):  FY  2004  Actual  %  of  Dollars  PBSA  >  35  %. 

Alternate  hypothesis  #2b  (Ha2b):  FY  2004  Actual  %  of  Dollars  PBSA  <  35%. 
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Because  the  data  represents  the  whole  population  of  PBSA  eligible  contracts,  if  during 
any  year  the  actual  percentage  of  eligible  dollars  awarded  using  PBSA  were  less  than  the 
interim  PBSA  goals  then  the  null  hypothesis  was  rejected. 

Investigative  Question  3.  Forecasting  using  simple  linear  regression  and  a  two- 
period  moving  average  were  used  in  order  to  answer  investigative  question  3.  Because 
the  number  of  data  points  (4)  was  limited,  the  researcher  opted  to  use  more  than  one 
forecasting  method.  Despite  the  use  of  two  forecasting  methods,  the  significance  of  the 
forecasts  was  expected  to  be  limited  due  to  the  few  data  points  available. 

With  simple  linear  regression  a  straight-line  model  is  used  to  relate  a  times  series, 
Y,,  to  time,  t  (McClave,  et  al.,  2001).  For  this  study  Y,  was  the  percentage  of  PBSA 
dollars  or  actions  and  t  was  the  FY.  Using  this  model,  a  least  squares  line  can  be 
calculated  and  used  to  forecast  future  values  of  Y,  (McClave,  et  al.,  2001).  The  straight- 
line  model  is  as  follows: 

E(  Y ,)  =  p0  +  Pfi 

Where  Po  is  the  intercept  and  Pi  is  a  population  parameter.  This  model  can  be 
fitted  using  the  method  of  least  squares,  where  the  sum  of  squared  errors  between  the 
predicted  and  actual  values  is  smaller  than  any  other  straight-line  model  (McClave,  et  ah, 
2001).  The  least  squares  model  looks  identical  to  the  straight-line  model,  except  for  least 
square  estimators  are  substituted  for  the  population  parameters.  The  least  square  model  is 
as  follows: 

E(Y,)=  Po+  M 
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Moving  averages  forecast  for  future  time  periods  using  the  mean  of  past 
observations  (Makridakis,  Wheelwright,  &  McGee,  1978).  With  a  two-period  moving 
average,  only  the  mean  of  the  last  two  observations  is  use  to  forecast  for  the  value  of  the 
next  time  period  (Makridakis,  Wheelwright,  &  McGee,  1978).  For  this  study,  the  mean 
of  the  percentages  for  FY  2003  and  2004  will  be  used  to  forecast  the  percentage  for  FY 
2005. 


Using  the  simple  liner  regression  model,  moving  averages,  and  the  following 
hypotheses,  the  researcher  forecasted  for  the  percentage  of  PBSA  dollars  and  actions  for 
FY  2005: 

Null  hypothesis  #3  (H03):  The  USAF  will  meet  the  mandated  FY  2005  goal  of 
50%  of  all  eligible  service  dollars  and  actions  awarded  using  PBSA. 

Alternate  hypothesis  #3  (Ha3):  The  USAF  will  not  meet  the  mandated  FY  2005 
goal  of  50%  of  all  eligible  service  dollars  and  actions  awarded  using  PBSA. 

Null  hypothesis  #3a  (H03a):  The  FY  2005  %  of  dollars  PBSA  >  50% 

Alternate  hypothesis  #3a  (Ha3a):  The  FY  2005  %  of  dollars  PBSA  <  50% 

Null  hypothesis  #3b  (Ho3b):  The  FY  2005  %  of  action  PBSA  >  50% 

Alternate  hypothesis  #3b  (Ha3b):  The  FY  2005  %  of  actions  PBSA  <  50% 

If  either  forecasted  percentage  of  dollars  or  actions  PBSA  for  FY  2005  was  less  than 

50%,  then  the  null  sub-hypothesis  was  rejected.  If  any  of  the  null  sub-hypotheses  were 

rejected  then  null  hypothesis  #4  was  also  rejected. 

Investigative  Question  4.  In  order  to  answer  investigative  question  4,  the 

following  hypotheses  were  established: 


Null  hypothesis  #4A  (H04  dollars):  The  percentage  of  modifications,  measured  in 
dollars,  coded  PBSA  is  equal  to  the  percentage  of  non-modifications,  measured  in 
dollars,  coded  PBSA. 

Alternate  hypothesis  #4A  (Ha4  dollars):  The  percentage  of  modifications,  measured 
in  dollars,  coded  PBSA  is  not  equal  to  the  percentage  of  non-modifications, 
measured  in  dollars,  coded  PBSA. 
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Null  hypothesis  #4B  (H04  actions):  The  percentage  of  modifications,  measured  in 
actions,  coded  PBSA  is  equal  to  the  percentage  of  non-modifications,  measured  in 
actions,  coded  PBSA. 

Alternate  hypothesis  #4B  (Ha4  actions):  The  percentage  of  modifications,  measured 
in  actions,  coded  PBSA  is  not  equal  to  the  percentage  of  non-modifications, 
measured  in  actions,  coded  PBSA. 

Because  the  whole  population  was  used  when  computing  the  percentages,  there  was  no 
need  to  use  a  statistical  test  to  detennine  a  difference  between  modifications  and  non¬ 
modifications.  The  researcher  decided  that  a  5  percent  difference  of  either  dollars  or 
actions  in  any  FY  was  significant.  Therefore,  the  decision  to  accept  or  reject  the  null 
hypothesis  was  determined  by  comparing  the  percentage  of  modifications  coded  PBSA  to 
the  percentage  of  non-modifications  coded  PBSA  for  FY  2001  through  FY  2004.  In 
order  to  make  this  comparison,  the  following  sub-hypotheses  were  established  for  each 
FY: 


Null  hypothesis  #4a  (H04a  2001  dollars):  |PBSA  Mod  %-PBSA  Non-mod  %|  <  5% 
Alternate  hypothesis  #4a  (Ha4a  2001  dollars):  |PBSA  Mod  %-PBSA  Non-mod  %|  >  5% 
Null  hypothesis  #4a  (H04a  2001  actions):  |PBSA  Mod  %-PBSA  Non-mod  %|  <  5% 
Alternate  hypothesis  #4a  (Ha4a  2001  actions):  |PBSA  Mod  %-PBSA  Non-mod  %|  >  5% 
Null  hypothesis  #4b  (HO4b2002doiiars):  |PBSA  Mod  %-PBSA  Non-mod  %|  <  5% 
Alternate  hypothesis  #4b  (Ha4b2002doiiars):  |PBSA  Mod  %-PBSA  Non-mod  %|  >  5% 
Null  hypothesis  #4b  (HO4b2002actionS):  |PBSA  Mod  %-PBSA  Non-mod  %|  <  5% 
Alternate  hypothesis  #4b  (Ha4b2002actions):  |PBSA  Mod  %-PBSA  Non-mod  %|  >  5% 
Null  hypothesis  #4C  (HO4C2003doiiars):  |PBSA  Mod  %-PBSA  Non-mod  %|  <  5% 
Alternate  hypothesis  #4C  (Ha4C2003doiiars):  |PBSA  Mod  %-PBSA  Non-mod  %|  >  5% 
Null  hypothesis  #4C  (HO4C2003actions):  |PBSA  Mod  %-PBSA  Non-mod  %|  <  5% 
Alternate  hypothesis  #4C  (Ha4C2003actions):  |PBSA  Mod  %-PBSA  Non-mod  %|  >  5% 
Null  hypothesis  #4d  (HO4d2004doiiars):  |PBSA  Mod  %-PBSA  Non-mod  %|  <  5% 
Alternate  hypothesis  #4d  (Ha4d2004doiiars):  |PBSA  Mod  %-PBSA  Non-mod  %|  >  5% 
Null  hypothesis  #4d  (HO4d2004actions):  |PBSA  Mod  %-PBSA  Non-mod  %|  <  5% 
Alternate  hypothesis  #4d  (Ha4d2004actionS):  |PBSA  Mod  %-PBSA  Non-mod  %|  >  5% 

If  the  difference  for  any  year  was  greater  than  5  percent,  then  the  null  hypothesis  for  that 

year  was  rejected.  If  any  of  the  null  sub-hypotheses  were  rejected  then  null  hypothesis  #4 

was  also  rejected. 
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Investigative  Question  5.  Contingency  tables  and  logistic  regression  were  used  to 
answer  investigative  question  5.  However,  before  running  any  analysis,  the  variables 
extracted  from  the  J001  database  were  evaluated  on  their  likeliness  of  producing 
significant  correlational  results.  Immediately,  three  variables  were  eliminated  from 
further  analysis  because  their  data  is  primarily  descriptive  in  nature.  These  variables 
were  Action  Date,  Completion  Date,  and  City  or  Place  Code.  All  remaining  variables 
were  analyzed. 

All  nominal  variables  were  tested  for  dependency  with  PBS  A  using  contingency 
tables.  Contingency  tables  are  a  statistical  model  used  for  multi-nomial  data  and  provide 
a  detennination  of  dependence  (McClave,  et  ah,  2001).  Therefore,  the  two  continuous 
variables  in  the  data  set,  Obligated  or  Deobligated  Dollars  and  Total  Estimated  Contract 
Value,  were  not  tested  using  this  statistical  tool.  Contingency  tables  provide  observed 
counts  of  occurrences,  expected  counts  of  occurrences,  frequencies  for  each  cell,  row, 
and  column,  probabilities  for  each  cell,  row,  and  column.  Contingency  tables  are 
constructed  by  listing  all  the  possible  outcomes  of  one  variable  as  rows  in  a  table  and  the 
possible  outcomes  of  the  other  variable  as  columns,  then  finding  the  frequency  and 
probability  for  each  cell.  The  cell  frequencies  and  probabilities  are  then  summed  across 
both  rows  and  columns.  The  sums  are  placed  in  the  margins,  the  values  of  which  are 
called  marginal  frequencies  and  marginal  probabilities  (McClave,  et  ah,  2001).  The  lower 
right  hand  corner  value  contains  the  sum  of  the  row  and  column  marginal  frequencies  and 
marginal  probabilities.  The  sum  of  the  marginal  frequencies  must  be  equal  to  N,  the  total 
number  of  trials,  and  the  sum  of  the  marginal  probabilities  must  be  equal  to  1  (McClave, 
et  ah,  2001).  Conservative  sources  maintain  that  cells  that  have  an  expected  value  of  5  or 
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less  must  be  removed  because  they  will  cause  inconsistencies  in  the  table  (McClave  et  al., 
2001).  However,  others  (Cochran,  1954)  maintain  that  data  does  not  need  to  be  removed 
as  along  as  at  least  80%  of  the  expected  cell  counts  are  5  or  more  and  no  expected  cell 
count  is  less  than  1 . 

The  chi-square  (y2)  statistic  is  a  test  statistic  used  to  analyze  count  data  (McClave, 

et  ah,  2001)  Using  x2,  a  determination  of  dependency  between  two  classifications  can  be 

made  (McClave,  et  al.,  2001).  For  this  study  the  two  classifications  were  the  contract 

variable  PBSA  and  a  particular  contract  variable  of  interest.  Each  contract  variable,  or 

classification,  was  tested  against  PBSA  in  order  to  detennine  dependency.  If  the  two 

variables  are  dependent,  knowing  one  should  increase  the  probability  of  knowing  the 

other.  In  order  to  test  for  dependency,  the  following  hypotheses  were  established: 

Null  hypothesis  #6  (H06):  Variable  “x”  and  PBSA  are  independent 
Alternate  hypothesis  #6  (Ha6):  Variable  “x”  and  PBSA  are  dependent 

The  test  was  conducted  for  each  nominal  variable.  However,  before  accepting  or 

rejecting  the  null  hypothesis,  a  chi-square  alpha  (x  „)  needed  to  be  established.  For  this 

test  an  alpha  of  .01  was  used.  The  next  thing  needed  in  determining  x  a  was  the  degrees 

of  freedom  (df).  The  following  equation  was  used  to  calculate  the  df  for  each  variable: 

df=(r-l)(c-l) 

Where  r  equals  the  number  of  rows  and  c  equals  the  number  of  columns  in  the 

2  2 

contingency  table.  Using  a  %  table,  a  x%  for  each  variable  was  established  based  on  each 
variable’s  df  and  an  a  of  .01.  Finally,  x  and  x  a  were  compared.  If  x  was  greater  than 
X2a  the  null  hypothesis  of  independence  was  rejected  and  the  alternate  hypothesis  of 
dependence  was  accepted.  (McClave,  et  al.,  2001) 


56 


Lastly,  it  is  important  to  note  that  while  contingency  tables  can  detennine 
dependency  between  two  classifications,  they  do  not  establish  the  extent  of  the 
dependency  or  the  existence  of  a  causal  relationship  between  the  two  classifications 
(McClave,  et  ah,  2001). 

The  continuous  variables  and  all  the  variables  that  were  found  to  be  dependent 
with  PBSA  using  contingency  tables  were  further  analyzed  using  logistic  regression. 
Logistic,  also  known  as  categorical,  regression  is  performed  when  data  is  both 
quantitative  and  qualitative  in  nature  and  the  dependent  variable  has  only  two  possible 
outcomes  (binary)  (McClave,  et  ah,  2001).  The  dependent  variable  in  this  study,  which  is 
whether  a  contract  action  is  PBSA,  is  qualitative  and  has  only  two  possible  outcomes, 

“Y”  for  yes  or  “N”  for  no.  It  is  common  to  use  the  tenns  success  and  failure  for  these 
types  of  binary  responses  (Agresti  and  Finlay,  1997).  With  the  exception  of  Obligated  or 
Deobligated  Dollars  and  Total  Estimated  Contract  Value,  all  other  independent  variables 
are  qualitative  and  can  be  described  in  binary  or  multi-nomial  form. 

When  the  dependent  variable  is  binary,  the  probability,  p,  that  the  dependent 
variable,  y,  is  response  level  j  is  estimated  by  dividing  the  total  sample  count,  n,  into  the 
total  of  each  response  level  nj,  and  is  written: 

pj=n/n 

This  model  serves  the  same  role  for  a  binary  response  as  the  sample  mean  does  for 
continuous  models.  (McClave,  et  ah,  2001) 

Ordinary  least  squares  regression  models  the  mean  of  the  response  variable.  With 
a  binary  response  variable,  the  model  describes  how  the  proportion  of  successes  depends 
on  the  independent  variables  (Agresti  and  Finlay,  1997).  Pi  (71)  usually  denotes  the  true 
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proportion  of  successes  and  also  represents  the  probability  that  a  randomly  selected 
subject  has  a  success  response  that  varies  according  to  the  dependent  variables  (Agresti 
and  Finlay,  1997).  The  logistic  regression  model  is: 

Log  (n  /( 1-  7t))  =  a  +  P  X  +  s 

Where  [n  /(  I  -  7t)]  is  the  odds  ratio,  a  is  the  intercept,  [3  is  the  parameter  of  the  dependent 
variable,  X,  and  s  is  the  error.  In  this  model,  as  n  increases  from  0  to  1,  the  odds  ratio 
increases  from  0  to  oo  and  the  logit  increases  from  -oo  to  oo  (Agresti  and  Finlay,  1997). 
Additionally,  a  n  value  greater  than  .5  has  a  positive  logit  value  and  n  values  less  than  .5 
have  a  negative  logit  value  (Agresti  and  Finlay,  1997). 

The  null  hypothesis  that  (3  =  0  and  the  independent  variable,  X,  has  no  effect  on 
the  probability,  n,  of  a  success,  can  be  tested  using  either  a  z  test  statistic,  which  is  the 
estimate  of  [3  divided  by  the  standard  error,  or  the  square  of  the  z  statistic,  which  is  called 
the  Wald  statistic  (Agresti  and  Finlay,  1997).  The  Wald  statistic  can  be  used  in  this 
instance  because  it  has  a  %  distribution  with  df  =  1  and  and  the  same  P-value  as  the  z 
statistic.  Rejection  of  the  null  hypothesis  results  in  acceptance  of  the  alternate  hypothesis 
that  P  ^  0.  In  order  to  test  each  variable  in  this  study,  the  following  hypotheses  were 
established: 

Null  hypothesis  #5  (H05):  The  parameter  for  variable  “x”  equals  zero. 

Alternate  hypothesis  #5  (Has):  The  parameter  for  variable  “x”  does  not  equal 

zero. 

2 

The  determination  to  accept  or  reject  the  null  hypothesis  was  based  on  each  computed  % 
and  corresponding  a  value.  For  this  test,  the  significance  level  was  set  at  .01. 
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Another  test  frequently  used  is  the  likelihood-ratio  test.  This  test  can  be  used  to 
compare  two  models  by  testing  that  the  extra  parameters  in  the  complete  model  equal 
zero  (Agresti  and  Finlay,  1997).  Negative  log-likelihood  is  the  negative  sum  of  logs  of 
the  observed  probabilities.  The  negative  log-likelihood  plays  the  same  role  as  sums  of 
squares  does  for  continuous  data,  but  for  categorical  data  (Agresti  and  Finlay,  1997). 

Twice  the  value  of  the  negative  log-likelihood  (-Xoglikelihood)  also  has  a  distribution  that 
is  approximately  a  y  distribution  (Agresti  and  Finlay,  1997).  Therefore,  the  y  test  can  be 

used  to  test  the  null  hypothesis  that  removing  all  the  variables  from  the  model  leaves  the 

2 

likelihood  of  observing  the  sample  unchanged.  This  test  is  analogous  to  the  F-test  for  R 
in  multiple  regression  which  tests  whether  or  not  the  improvement  in  the  model 
associated  with  the  additional  variables  is  statistically  significant  (Agresti  and  Finlay, 
1997).  If  we  are  able  to  reject  the  null  hypothesis,  we  have  evidence  that  at  least  one  of 
the  variables  does  have  an  effect  on  the  response. 

R  (U),  the  uncertainty  coefficient,  measures  the  total  uncertainty  that  is  attributed 
to  the  model  and  is  calculated  using  the  following  formula: 

2  /t  T\  _  -log  likelihood  (difference) 

R  (U)  —  - 

-  log  likelihood  (reduced) 

Negative  log-likelihood  (difference)  is  the  difference  between  the  likelihood  using  the 
full  model  and  the  likelihood  using  a  model  with  no  variables.  Negative  log-likelihood 
(reduced)  is  likelihood  using  no  variables  in  the  model.  R“  (U)  values  must  be  between  0 
and  1 .  An  R"  (U)  of  1  means  that  the  factor  completely  predicts  the  categorical  response; 
an  R“  (U)  of  0  means  there  is  no  gain  by  using  the  model  over  using  fixed  responses 
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(McClave,  et  al.,  2001).  It  is  important  to  note  that  high  R“  (U)  values  are  rare  in 
categorical  models  (JMP,  2003). 

Validity  and  Reliability 

Validity  and  reliability  of  a  measurement  instrument  influence  the  probability  that 
you  will  obtain  statistical  significance  in  your  data  analysis,  and  the  extent  to  which  you 
can  draw  meaningful  conclusions  from  your  data  (Leedy  and  Ormrod,  2001).  While 
efforts  were  made  to  make  this  study  as  valid  and  reliable  as  possible,  there  are  still  some 
validity  and  reliability  concerns. 

The  validity  pertains  to  the  accuracy,  meaningfulness,  and  credibility  of  the 
research  effort  as  a  whole  (Leedy  and  Ormrod,  2001).  Most  texts  separate  validity  into 
two  categories;  internal  and  external. 

Internal  validity  is  the  extent  to  which  a  researcher  can  draw  an  accurate 
conclusion  about  relationships  within  the  data  based  on  the  research  design  (Leedy  and 
Ormrod,  2001).  Because  the  data  from  the  J001  was  not  personally  gathered,  but  rather 
queried  from  an  existing  database,  there  is  little  that  can  be  done  to  counter  any  internal 
validity  concerns,  the  largest  being  construct  validity.  Construct  validity  is  the  extent  to 
which  an  instrument  measures  a  characteristic  that  cannot  be  directly  observed,  but  must 
be  inferred  (Leedy  and  Ormrod,  pg  98).  Personal  experience  by  the  researcher  suggests 
that  time  constraints,  pressure  to  meet  procurement  goals,  and  a  general  lack  of  emphasis 
on  proper  coding  may  attribute  to  inaccuracies  in  PBSA  coding  in  the  J001.  Although 
this  study  does  not  measure  these  influences,  it  does  test  and  analyze  the  extent  of 
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possible  validity  concerns  by  evaluating  a  sample  of  PBS  A  coded  contracts  against  the 
PBS  A  criteria  identified  in  the  PBS  A  Guidebook. 

External  validity  is  the  extent  to  which  a  study’s  results  apply  to  situations  beyond 
the  study  itself  (Leedy  and  Ormrod,  2001).  Two  areas  of  this  study  have  external  validity 
concerns.  First,  the  contracts  evaluated  from  the  three  WPAFB  contracting  offices 
represent  only  a  sample  of  PBSA  coded  contracts.  While  this  study  does  not  contend  that 
the  sampled  PBSA  contracts  from  these  three  offices  are  a  proper  representation  of  all 
PBSA  contracts  in  the  USAF,  the  study  does  maintain  that  the  contracts  evaluated  are  a 
reasonable  representation  and  sample  of  all  the  PBSA  coded  contracts  in  those  respective 
offices.  Secondly,  because  only  the  services  above  $100,000  not  excluded  from  PBSA 
use  were  analyzed  in  this  study,  results  and  conclusions  drawn  from  this  study  do  not 
apply  to  services  less  than  or  equal  to  $100,000  or  services  excluded  from  PBSA  use. 
Although  services  falling  in  these  categories  are  not  required  to  be  procured  using  PBSA 
techniques,  these  techniques  are  encouraged  and  sometimes  used  when  it  is  deemed 
applicable.  How  and  if  these  actions  differ  from  eligible  actions  is  unknown.  However, 
because  many  of  these  actions  are  relatively  insignificant  in  dollar  amount  compared  to 
the  entire  sum  of  the  actions,  it  is  assumed  they  would  have  little  to  no  effect  on  the 
cumulative  calculations. 

The  reliability  of  a  measurement  instrument  is  the  extent  to  which  it  yields 
consistent  results  when  measuring  characteristics  that  have  not  changed  (Feedy  and 
Ormrod,  2001).  The  proven  analysis  tools  of  logistic  regression  and  contingency  tables 
do  not  hinder  the  reliability  of  the  study,  but  there  are  some  reliability  concerns  with  the 
data  in  the  J001  database.  The  two  previously  mentioned  GAO  reports  alluded  to  these 
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reliability  concerns  within  all  government  procurements  databases.  While  it  is 
impossible  to  counter  any  subjective  decisions  incorrectly  made  by  individuals  inputting 
data  into  the  database,  it  is  possible  to  assess  the  reliability,  which  is  done  by  sampling 
PBS  A  coded  contracts.  However,  because  the  sample  only  looks  at  PBS  A,  the  reliability 
of  other  DD350  characteristics  is  not  accounted  for. 

Summary 

Chapter  3  discusses  the  research  problem,  investigative  questions,  data  gathering, 
data  reliability,  data  analysis,  and  validity  and  reliability.  Chapter  4  will  address  the 
results  and  analysis,  findings  based  on  the  results  and  analysis,  and  any  conclusions  that 
can  be  drawn  from  the  findings. 
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IV.  Results  and  Analysis 


Introduction 

This  chapter  describes  the  research  results  and  answers  to  each  of  the  five 
investigative  questions.  Referenced  throughout  the  chapter  are  multiple  appendices  that 
provide  more  detailed  breakouts  of  the  research  results.  Included  in  these  appendices  are 
spreadsheets  that  separate  PBSA  actions  and  dollars  by  Service  Category,  Type  of 
Contract,  and  Type  of  Entity.  These  appendices  can  be  found  at  the  end  of  this  thesis. 

Data  Analysis 

Investigative  Question  1.  In  order  to  answer  investigative  question  1,  the 

following  hypotheses  were  established: 

Null  hypothesis  #1  (H0i):  p  >  .  10 
Alternate  hypothesis  #1  (Hai):  p  <  .10 

Although  the  goal  was  to  evaluate  only  30  PBSA  contracts  from  each  of  the  three  offices, 
the  researcher  managed  to  evaluate  a  total  of  102  contracts;  36  from  AFRL  Det  1/PK,  34 
from  88ABW/PK,  and  32  from  ASC/PK.  Of  the  102  contracts  evaluated,  3  were 
determined  not  to  meet  the  minimum  mandatory  criteria  identified  in  the  PBSA 
Guidebook.  Therefore,  the  z  test  statistic  was  calculated  using  n  =  102  and  p  =  3/102,  or 
0.029.  Substituting  the  values  into  the  test  statistic,  a  z  value  of  -2.38  was  obtained. 

Test  Statistic:  z  =  — — — 
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.029 -.1 


(-1X-9) 

V  102 

=  -2.38 

Because  the  calculated  z  value  is  less  than  -2.33,  which  is  the  cut-off  for  the 

rejection  region,  the  null  hypothesis  is  rejected.  Therefore,  there  is  sufficient  evidence  at 

the  .01  level  of  significance  to  indicate  that,  for  the  three  offices  sampled,  fewer  than 

10%  of  the  PBSA  contracts  were  coded  incorrectly.  Even  though  this  test  can  only  be 

generalized  to  the  three  offices  the  samples  were  drawn  from,  the  results  indicate  that 

there  may  not  be  systematic  problems  in  PBSA  coding  throughout  the  USAF. 

Investigative  Question  2.  In  order  to  answer  investigative  question  2,  the 

following  hypotheses  were  established: 

Null  hypothesis  #2  (H02):  The  USAF  is  meeting  interim  PBSA  goals. 

Alternate  hypothesis  #2  (Ha2):  The  USAF  is  not  meeting  interim  PBSA  goals. 

Additionally,  the  following  sub-hypotheses  were  established  in  order  to  assist  in 

answering  investigative  question  2. 

Null  hypothesis  #2a  (H02a):  FY  2003  Actual  %  of  Dollars  PBSA  >  25  %. 
Alternate  hypothesis  #2a  (Ha2a):  FY  2003  Actual  %  of  Dollars  PBSA  <  25%. 

Null  hypothesis  #2b  (Ho2b):  FY  2004  Actual  %  of  Dollars  PBSA  >  35  %. 
Alternate  hypothesis  #2b  (Ha2b):  FY  2004  Actual  %  of  Dollars  PBSA  <  35%. 

Table  3  displays  that  in  Fiscal  Year  (FY)  2003  the  USAF  exceeded  the  interim  goal  of 
25%  by  awarding  over  32%  of  its  contracts  using  PBSA,  a  difference  of  over  7%. 
However,  for  FY  2004  the  USAF  awarded  only  28.75%  of  its  contracts  using  PBSA; 
when  compared  to  the  goal  of  35%,  the  USAF  was  over  6%  below  the  interim  goal. 
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Table  3.  The  Percentage  of  Dollars  PBSA  Actual  Compared  to  the  Percentage  of  Dollars 
PBSA  Goal  for  Fiscal  Years  (FY)  2003  and  2004. 


Percentage  of  Dollars  PBSA 

FY03 

FY04 

Actual 

32.27 

28.75 

Goal 

25 

35 

Difference 

7.27 

(6.25) 

The  results  conclude  that  the  USAF  did  not  meet  the  FY  2004  goal,  and  is  not  meeting 
interim  PBSA  goals.  Therefore,  null  hypothesis  #2  is  rejected. 

These  numbers  are  better  illustrated  using  Figure  1  below.  Included  in  this  figure 
is  the  percentage  of  dollars  PBSA  from  FY  2001  through  FY  2004,  as  well  as  the  50% 
PBSA  goal  for  FY  2005.  For  the  first  three  years  there  was  a  distinct  upward  trend  in  the 
percentage  of  actual  dollars  awarded  using  PBSA.  However,  in  FY  2004  the  trend  was 
broken  and  there  was  a  noticeable  drop  in  the  actual  PBSA  percentage. 
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Figure  1.  The  percentage  of  actual  dollars  PBSA  and  the  goal  for  percentage  of  dollars 
PBSA  as  a  function  of  Fiscal  Year. 


Further  analyzing  the  data  using  the  summary  tables  included  in  Appendices  C,  D, 
E,  and  F,  additional  interesting  phenomena  were  discovered.  For  instance,  Figure  2 
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shows  that  during  FY  2004,  not  only  did  the  percentage  of  PBSA  dollars  decline,  but  so 
did  the  cumulative  PBSA  dollars  and  cumulative  total  dollars.  Thus,  PBSA  awards 
included  a  smaller  percentage  of  a  smaller  total  dollars  awarded.  Although  it  is  uncertain 
what  caused  this  phenomenon,  it  may  be  associated  with  the  reason  for  a  decrease  in  the 
PBSA  percentage  during  FY  2004. 
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Figure  2.  PBSA  dollars  and  total  dollars  as  a  function  of  Fiscal  Year. 


Additionally,  when  analyzing  the  data  by  variable  Type  of  Contract,  it  is 
interesting  to  note  that  contracts  coded  L  (Fixed  Price  Incentive),  M  (Fixed  Price  Award 
Fee),  R  (Cost  Plus  Award  Fee),  Y  (Time  and  Materials),  and  Z  (Labor  Hour)  or  left 
blank,  all  had  a  decrease  in  the  percentage  of  dollars  PBSA  for  FY  2004  (see  Figure  3). 
The  decrease  in  Cost  Plus  Award  Fee  contracts  is  the  most  significant  because  in  FY 
2004,  these  types  of  contracts  accounted  for  over  25%  of  the  total  eligible  dollars.  All  the 
other  contracts  that  realized  a  decrease  in  the  percentage  of  dollars  PBSA  for  FY  2004 
each  accounted  for  less  than  9%  of  the  total  eligible  dollars. 
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Figure  3.  Percentage  of  total  dollars  PBSA  by  Type  of  Contract. 


Surprisingly,  contracts  coded  J  (Firm  Fixed  Price)  and  U  (Cost  Plus  Fixed  Fee), 
which  together  account  for  over  45%  of  all  eligible  dollars  during  FY  2004,  both 
managed  to  increase  their  percentage  of  dollars  PBSA.  Unfortunately,  these  increases 
could  not  overcome  the  effect  of  the  percentage  decreases  in  the  other  contract  types. 

Investigative  Question  3.  In  order  to  answer  investigative  question  3,  the 
following  hypotheses  were  established: 

Null  hypothesis  #3  (H03):  The  USAF  will  meet  the  mandated  FY  2005  goal  of 
50%  of  all  eligible  service  dollars  and  actions  awarded  using  PBSA. 

Alternate  hypothesis  #3  (Ha3):  The  USAF  will  not  meet  the  mandated  FY  2005 
goal  of  50%  of  all  eligible  service  dollars  and  actions  awarded  using  PBSA. 

Null  hypothesis  #3a  (Ho3a):  The  FY  2005  %  of  dollars  PBSA  >  50% 

Alternate  hypothesis  #3a  (Ha3a):  The  FY  2005  %  of  dollars  PBSA  <  50% 

Null  hypothesis  #3b  (H03b):  The  FY  2005  %  of  action  PBSA  >  50% 

Alternate  hypothesis  #3b  (Ha3b):  The  FY  2005  %  of  actions  PBSA  <  50% 
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The  JMP  outputs  in  Figures  4  and  5  summarize  the  least  squares  fit  of  the  linear 


regression  models  to  the  percentage  dollars  and  actions  PBSA. 


Summary  of  Fit 

RSquare  0.367858 

RSquareAdj  0.051787 

Root  Mean  Square  Error  2.678479 

Mean  of  Response  28.89007 

Observations  (or  Sum  Wgts)  4 

Analysis  of  Variance 


Source 

Model 

Error 

C.  Total 

DF  Sum  of  Squares 

1  8.349715 

2  14.348495 

3  22.698209 

Mean  Square  F  Ratio 

8.34971  1.1638 

7.17425  Prob  >  F 

0.3935 

Parameter  Estimates 

Term 

Estimate  Std  Error 

t  Ratio 

Prob>[t| 

Intercept 

25.659414  3.280453 

7.82 

0.0160 

FY 

1.2922627  1.197852 

1.08 

0.3935 

Figure  4.  JMP  printout  of  least  squares  fit  to  percentage  of  dollars  PBSA. 


Summary  of  Fit 

RSquare  0.788726 

RSquare  Adj  0.683089 

Root  Mean  Square  Error  3.295191 

Mean  of  Response  28.58397 

Observations  (or  Sum  Wgts)  4 

Analysis  of  Variance 


Source  DF  Sum  of  Squares 

Model  1  81.07211 

Error  2  21.71657 

C.  Total  3  102.78868 

Parameter  Estimates 

Term  Estimate  Std  Error  t  Ratio  Prob>|t| 

Intercept  18.517191  4.035769  4.59  0.0444 

FY  4.0267135  1.473654  2.73  0.1119 

Figure  5.  JMP  printout  of  least  squares  fit  to  percentage  of  actions  PBSA. 


Mean  Square  F  Ratio 
81.0721  7.4664 

10.8583  Prob  >  F 
0.1119 
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As  expected,  neither  model  was  very  predictive,  as  evident  by  the  low  F  statistic 
values  and  high  P-values.  The  percentage  drop  for  FY  2004  and  the  use  of  only  four  data 
points  to  build  these  models,  both  had  a  negative  effect  on  the  establishment  of  a  good 
trend  line.  However,  despite  the  models  limitations,  they  were  still  used  to  forecast  for 
the  percentage  of  dollars  and  actions  PBS  A  for  FY  2005. 

Using  the  calculated  intercept  and  parameter  coefficients,  the  following  models 
were  established: 

%  Dollars  PBSA  =  25.6594  +  1.2922  *  Year 
%  Actions  PBSA  =  18.5172  +  4.0267  *  Year 

By  substituting  5  in  for  the  variable  Year,  the  following  forecasted  PBSA  percentages 

were  calculated: 

%  Dollars  PBSA  =  32.12 
%  Actions  PBSA  =  38.65 

Figures  6  and  7  provide  a  better  illustration  of  the  linear  regression  lines  plotted  with  the 
actual  PBSA  percentages. 
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Figure  6.  The  percentage  of  dollars  PBSA  and  the  liner  regression  line  for  percentage  of 
dollars  PBSA  as  a  function  of  Fiscal  Year. 
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Figure  7.  The  percentage  of  actions  PBSA  and  the  liner  regression  line  for  percentage  of 
actions  PBSA  as  a  function  of  Fiscal  Year. 
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Obviously  the  PBSA  percentages  for  both  dollars  and  actions  are  well  below  the 
goal  of  50%,  thus  resulting  in  the  rejection  of  the  null  hypotheses  for  both  PBSA  dollars 
and  actions. 

Assuming  the  FY  2004  percentages  were  anomalies,  the  FY  2004  data  points 
were  removed  and  new  regression  models  were  built.  The  JMP  outputs  in  Figures  8  and 
9  summarize  the  least  squares  fit  of  the  linear  regression  models  to  the  percentage  dollars 
and  actions  PBSA,  excluding  FY  2004. 


Summary  of  Fit 

RSquare  0.999649 

RSquare  Adj  0.999297 

Root  Mean  Square  Error  0.089255 

Mean  of  Response  28.93533 

Observations  (or  Sum  Wgts)  3 

Analysis  of  Variance 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Model 

1 

22.665665 

22.6657 

2845.16 

Error 

1 

0.007966 

0.0080 

Prob  >  F 

C.  Total 

2 

22.673631 

0.0119 

Parameter  Estimates 


Term  Estimate  Std  Error  t  Ratio  Prob>|t| 

Intercept  22.202473  0.136339  162.85  0.0039 

FY  3.3664272  0.063113  53.34  0.0119 

Figure  8.  JMP  printout  of  least  squares  fit  to  percentage  of  dollars  PBSA,  excluding  FY 
2004. 
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Summary  of  Fit 

RSquare  0.995859 

RSquareAdj  0.991718 

Root  Mean  Square  Error  0.59805 

Mean  of  Response  27.4144 

Observations  (or  Sum  Wgts)  3 

Analysis  of  Variance 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Model 

1 

86.016080 

86.0161 

240.4945 

Error 

1 

0.357663 

0.3577 

Prob  >  F 

C.  Total 

2 

86.373743 

0.0410 

Parameter  Estimates 


Term  Estimate  Std  Error  t  Ratio  Prob>|t| 

Intercept  14.298294  0.913536  15.65  0.0406 

FY  6.5580515  0.422885  15.51  0.0410 

Figure  9.  JMP  printout  of  least  squares  fit  to  percentage  of  actions  PBSA,  excluding  FY 
2004. 


Removing  the  data  point  for  FY  2004  significantly  increased  the  reliability  of  the 
models,  as  evident  by  the  higher  F  statistic  values  and  lower  P-values.  Using  these 
models,  the  percentage  of  dollars  and  actions  PBSA  for  FY  2005  were  calculated. 

Using  the  calculated  intercept  and  parameters  coefficients,  the  following  models 
were  established: 

%  Dollars  PBSA  =  22.2025  +  3.3664  *  Year 
%  Actions  PBSA  =  14.2983  +  6.5580  *  Year 

By  substituting  5  in  for  the  variable  Year,  the  following  forecasted  PBSA  percentages 

were  calculated: 

%  Dollars  PBSA  =  39.03 
%  Actions  PBSA  =  47.09 

Again,  the  PBSA  percentages  for  both  dollars  and  actions  are  below  the  goal  of  50%, 
thus  resulting  in  the  rejection  of  the  null  hypotheses  for  both  PBSA  dollars  and  actions. 
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A  2-period  moving  average  was  also  used  to  forecast  for  FY  2005  PBSA 
percentages.  Using  this  method,  the  following  forecasted  percentages  for  FY  2005  were 
calculated: 

%  Dollars  PBSA  =  30.51 

%  Actions  PBSA  =  32.91 

Assuming  the  FY  2004  percentages  were  anomalies,  the  FY  2004  data  points 
were  removed  and  new  percentages  were  calculated  for  FY  2004  and  2005  using  a  2- 
period  moving  average.  Using  this  method  the  following  forecasted  percentages  for  FY 
2004  and  2005  were  calculated: 

FY  2004  %  Dollars  PBSA  =  30.64 

FY  2004  %  Actions  PBSA  =  30.82 

FY  2005  %  Dollars  PBSA  =  30.54 

FY  2005  %  Actions  PBSA  =  32.39 

The  removal  of  the  FY  2004  percentages  resulted  in  only  a  small  change  in  the 
forecasted  percentages  for  FY  2005.  Again,  the  PBSA  percentages  for  both  dollars  and 
actions  are  below  the  goal  of  50%,  thus  resulting  in  the  rejection  of  the  null  hypotheses 
for  both  PBSA  dollars  and  actions.  Table  4  summarizes  all  forecast  results  for  FY  2005. 
Table  4.  Summary  of  Forecast  Results. 


Forecasting  Method 

FY  2005  Forecast 

%  Actions  PBSA%  Dollars  PBSA 

Linear  Regression 

38.65 

32.12 

Linear  Regression  (w/out  FY  2004) 

47.09 

39.03 

Moving  Average 

32.91 

30.51 

Moving  Average  (w/out  FY  2004) 

32.39 

30.54 

All  forecasted  PBSA  percentages  for  FY  2005,  measured  in  both  dollars  and  actions, 
were  below  the  goal  of  50%  and  resulted  in  the  rejection  of  the  null  hypothesis. 
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Investigative  Question  4.  In  order  to  answer  investigative  question  4,  the 

following  hypotheses  were  established: 

Null  hypothesis  #4  (H04  dollars):  The  percentage  of  modifications,  measured  in 
dollars,  coded  PBSA  is  equal  to  the  percentage  of  non-modifications,  measured  in 
dollars,  coded  PBSA. 

Alternate  hypothesis  #4  (Ha4  dollars):  The  percentage  of  modifications,  measured  in 
dollars,  coded  PBSA  is  not  equal  to  the  percentage  of  non-modifications, 
measured  in  dollars,  coded  PBSA. 

Null  hypothesis  #4  (H04  actions):  The  percentage  of  modifications,  measured  in 
actions,  coded  PBSA  is  equal  to  the  percentage  of  non-modifications,  measured  in 
actions,  coded  PBSA. 

Alternate  hypothesis  #4  (Ha4  acti0ns):  The  percentage  of  modifications,  measured 
in  actions,  coded  PBSA  is  not  equal  to  the  percentage  of  non-modifications, 
measured  in  actions,  coded  PBSA. 

Additionally,  the  following  sub-hypotheses  were  established  in  order  to  assist  in 
answering  investigative  question  4: 


Null  hypothesis  #4a  (H04a  2001  dollars):  |PBSA  Mod  %-PBSA  Non-mod  %|  <  5% 
Alternate  hypothesis  #4a  (Ha4a  2001  dollars):  |PBSA  Mod  %-PBSA  Non-mod  %|  >  5% 
Null  hypothesis  #4a  (H04a  2001  actions):  |PBSA  Mod  %-PBSA  Non-mod  %|  <  5% 
Alternate  hypothesis  #4a  (Ha4a  2001  actions):  |PBSA  Mod  %-PBSA  Non-mod  %|  >  5% 
Null  hypothesis  #4b  (Ho4b2002doiiars):  |PBSA  Mod  %-PBSA  Non-mod  %|  <  5% 
Alternate  hypothesis  #4b  (Ha4b2oo2doiiars):  |PBSA  Mod  %-PBSA  Non-mod  %|  >  5% 
Null  hypothesis  #4b  (Ho4b2002actions):  |PBSA  Mod  %-PBSA  Non-mod  %|  <  5% 
Alternate  hypothesis  #4b  (Ha4b2002actions):  |PBSA  Mod  %-PBSA  Non-mod  %|  >  5% 
Null  hypothesis  #4C  (HO4C2003doiiars):  |PBSA  Mod  %-PBSA  Non-mod  %|  <  5% 
Alternate  hypothesis  #4C  (Ha4C2003doiiars):  |PBSA  Mod  %-PBSA  Non-mod  %|  >  5% 
Null  hypothesis  #4C  (Ho4c2003actions):  |PBSA  Mod  %-PBSA  Non-mod  %|  <  5% 
Alternate  hypothesis  #4C  (Ha4C2003actionS):  |PBSA  Mod  %-PBSA  Non-mod  %|  >  5% 
Null  hypothesis  #4d  (HO4d2004doiiarS):  |PBSA  Mod  %-PBSA  Non-mod  %|  <  5% 
Alternate  hypothesis  #4d  (Ha4d2004doiiars):  |PBSA  Mod  %-PBSA  Non-mod  %|  >  5% 
Null  hypothesis  #4d  (Ho4d2004actions):  |PBSA  Mod  %-PBSA  Non-mod  %|  <  5% 
Alternate  hypothesis  #4d  (Ha4d2004actionS):  |PBSA  Mod  %-PBSA  Non-mod  %|  >  5% 

A  summary  of  the  results  broken  out  for  each  year  in  both  dollars  and  actions  is  displayed 

in  Tables  5  and  6  below. 
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Table  5.  The  Percentage  of  Modification  Dollars  PBSA  Compared  to  the  Percentage  of 
Non-Modification  Dollars  PBSA  for  Fiscal  Years  (FY)  2001,  2002,  2003  and  2004. 


Percentage  of  PBSA  Dollars 

FY01 

FY02 

FY03 

FY04 

Modifications 

25.51 

25.60 

29.31 

22.32 

Non-Modifications 

25.56 

32.77 

34.57 

35.08 

Difference  (Absolute  Value) 

0.04 

7.17 

5.26 

12.75 

Table  6.  The  Percentage  of  Modification  Actions  PBSA  Compared  to  the  Percentage  of 


Non-Modification  Actions  PBSA  for 

Fiscal  Years  (FY)  2001,  2002,  2003  and  2004. 

Percentage  of  PBSA  Actions 

FY01 

FY02 

FY03 

FY04 

Modifications 

21.55 

27.96 

33.61 

32.45 

Non-Modific  ations 

20.17 

27.88 

33.78 

31.92 

Difference  (Absolute  Value) 

1.39 

0.08 

0.17 

0.54 

These  calculations  are  better  illustrated  using  line  graphs.  Figure  10  shows  that 
the  percentage  of  modifications  coded  PBSA,  measured  in  actions,  was  almost  identical 
to  the  percentage  of  non-modifications  coded  PBSA.  Figure  1 1  shows  that  the 
percentage  of  modifications  coded  PBSA,  measured  in  dollars,  was  more  than  5%  less 
than  the  percentage  of  non-modifications  coded  PBSA  for  FYs  2002,  2003,  and  2004. 
Therefore,  the  null  hypothesis  in  dollars  is  rejected,  but  the  null  hypothesis  in  actions 
cannot  be  rejected. 
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Figure  10.  The  percentage  of  modifications  PBSA  and  percentage  of  non-modifications 
PBSA,  measured  in  actions,  as  a  function  of  Fiscal  Year. 
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Figure  11.  The  percentage  of  modifications  PBSA  and  percentage  of  non-modifications 
PBSA,  measured  in  dollars,  as  a  function  of  Fiscal  Year. 
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Further  analysis  of  the  data  revealed  additional  interesting  phenomena.  Figure  12 
illustrates  how  the  dollar  amount  per  PBSA  modification  has  decreased  each  year,  while 
the  dollar  amount  per  non-PBSA  modifications  has  increased  the  last  three  years. 
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Figure  12.  The  dollars  per  action  for  all  PBSA  modifications  and  non-PBSA 
modifications,  as  a  function  of  Fiscal  Year. 

Conversely,  Figure  13  illustrates  how  the  dollar  amount  per  PBSA  non¬ 
modification,  which  are  new  PBSA  contracts,  has  been  greater  than  non-PBSA  non¬ 
modifications  each  year. 
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Figure  13.  The  dollars  per  action  for  all  PBS  A  non-modifications  and  non-PBSA  non¬ 
modifications,  as  a  function  of  Fiscal  Year. 

Investigative  Question  5.  The  following  hypotheses  were  established  in  order  to 
answer  investigative  question  5: 

Null  hypothesis  #6  (Ho6):  Classification  “x”  and  PBSA  are  independent 
Alternate  hypothesis  #6  (Ha6):  Classification  “x”  and  PBSA  are  dependent 

Dependency  was  detennined  using  contingency  tables.  All  classifications,  except  one, 

were  found  to  be  dependent,  thus  supporting  the  rejection  of  the  null  hypothesis.  Multi- 

Year  Contract  was  the  only  classification  that  yielded  a  chi-square  (%2)  value  that  was 

smaller  than  chi-square  alpha  (y  a),  thus  not  falling  in  the  rejection  region.  Again,  it  is 

important  to  note  that  contingency  tables  were  unable  to  determine  dependency  for  some 

classifications  because  expected  cell  counts  were  less  than  1 .  These  classifications 

included  FSC  or  SVC  Code,  NAICS  Code,  and  Authority  for  Other  than  Full  and  Open 

Competition.  Table  7  summarizes  the  results  from  the  contingency  table  analysis. 

Printouts  of  all  the  contingency  tables  can  be  found  in  Appendix  G. 
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Table  7.  Summary  of  Contingency  Tables 


~X~a  df  Prob> 
_ _ 


Classification 


Results 


B1C  -  Bundled  Contract 

BID  -  Bundled  Contract  Exception 

BIO  -  Multi-Year  Contract 

B13A  -  Contract/Order 

B13B  -  Type  of  Indefinite-Delivery 

Contract 

B13E  -  Multiple  Award  Contract  Fair 
Opportunity 

B13F  -  Indefinite-Delivery  Contract  Use 
Cl  -  Synopsis 

C2  -  Reason  Not  Synopsized 
C3  -  Extent  Competed 
C5  -  Type  of  Contract 
C8  -  Solicitation  Procedures 
C14  -  Commercial  Items 
D1A  -  Type  of  Entity 
DIB  -  Women-Owned  Business 
D1C  -  HUBZone  Representation 
DID  -  Ethnic  Group 
D4A  -  Type  of  Set-Aside 
D4B  -  Type  of  Preference 
D7  -  SBIR  Program 


124.58 

6.64 

1 

<.0001 

Dependent 

230.56 

11.34 

3 

<.0001 

Dependent 

0.07 

6.63 

1 

.7877 

Independent 

1790.86 

18.48 

7 

0.0000 

Dependent 

122.42 

11.34 

3 

<.0001 

Dependent 

10017.72 

15.09 

5 

0.0000 

Dependent 

2112.51 

13.28 

4 

0.0000 

Dependent 

1479.44 

11.34 

3 

<.0001 

Dependent 

603.47 

13.28 

4 

<.0001 

Dependent 

796.09 

13.28 

4 

<.0001 

Dependent 

4549.71 

26.22 

12 

<.0001 

Dependent 

3405.04 

21.67 

9 

0.0000 

Dependent 

2816.34 

9.21 

2 

0.0000 

Dependent 

2334.19 

24.73 

11 

0.0000 

Dependent 

1089.73 

11.34 

3 

<.0001 

Dependent 

563.87 

9.21 

2 

<.0001 

Dependent 

397.33 

18.48 

7 

<.0001 

Dependent 

1304.75 

20.09 

8 

<.0001 

Dependent 

1066.71 

11.34 

3 

<.0001 

Dependent 

1275.33 

13.28 

4 

<.0001 

Dependent 

For  single  response  variables,  such  as  Bundled  Contract,  a  result  of  “Dependent” 
means  that  there  is  dependence  between  the  variable  being  coded  “Yes”  or  “No”  and 
PBSA  being  coded  “Yes”  or  “No.”  For  multi-response  variables,  such  as  Contract/Order, 
a  result  of  “Dependent”  means  that  there  is  dependence  between  the  variable’s  possible 
responses  and  PBSA  being  coded  “Yes”  or  “No.”  However,  contingency  tables  only 
determine  dependency  between  two  classifications,  not  the  extent  of  the  relationship. 
Because  of  this,  contingency  tables  were  used  to  help  narrow  down  the  classifications 
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used  for  logistic  regression.  Unfortunately,  only  one  classification,  Multi-Year  Contract, 
was  independent,  thus  excluded  from  further  analysis. 

Logistic  regression  was  also  used  to  answer  investigative  question  5.  The 
following  hypotheses  were  established: 

Null  hypothesis  #5  (Ho5):  The  parameter  for  variable  “x”  equals  zero. 

Alternate  hypothesis  #5  (Has):  The  parameter  for  variable  “x”  does  not  equal 

zero. 

As  depicted  in  Table  8,  logistic  regression  yielded  the  same  x2  values  calculated 
using  contingency  tables.  This  was  expected  because  the  logistic  regression  models  were 
built  using  only  one  independent  variable  and  a  dependent  variable,  PBSA.  This  is 
similar  to  contingency  tables,  which  test  the  dependency  between  only  two  variables. 
Based  on  the  'i  and  associated  P-values,  the  null  hypotheses  for  all  variables,  except 
Obligated  or  Deobligated  Dollars,  are  rejected.  The  null  hypothesis  for  Obligated  or 
Deobligated  Dollars  is  not  rejected,  because  the  P-value  of  .9928  is  clearly  larger  than  the 
significance  level  of  .01. 
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Table  8.  Summary  of  Logistic  Regression 


Variable 

— 7 — 

Prob>  'i 

R2(U) 

B1C  -  Bundled  Contract 

124.58 

<.0001 

B8  -  Obligated  or  Deobligated  Dollars 

.000081 

0.9928 

B13A  -  Contract/Order 

1790.86 

0.0000 

B13B  -  Type  of  Indefinite -Delivery  Contract 

122.42 

<.0001 

B13E  -  Multiple  Award  Contract  Fair  Opportunity 

10017.72 

0.0000 

B13F  -  Indefinite-Delivery  Contract  Use 

212.51 

0.0000 

C 1  -  Synopsis 

1479.44 

<.0001 

C3  -  Extent  Competed 

796.09 

<.0001 

C5  -  Type  of  Contract 

4549.71 

0.0000 

C14  -  Commercial  Items 

28.16 

0.0000 

DIB  -  Women-Owned  Business 

1089.73 

<.0001 

D1C  -  HUBZone  Representation 

563.87 

<.0001 

DID  -  Ethnic  Group 

397.33 

<.0001 

D4A  -  Type  of  Set-Aside 

1304.75 

<.0001 

D4B  -  Type  of  Preference 

1066.71 

<.0001 

D7  -  SBIR  Program 

1275.33 

<.0001 

2  ...  9 

In  addition  to  %  and  associated  P-values,  logistic  regression  also  calculated  R“ 

(U)  values  for  each  variable.  As  shown  in  Table  8,  the  highest  R~  (U)  value  obtained  was 
.1081  for  Multiple  Award  Contract  Fair  Opportunity,  which  also  yielded  the  highest  %2 
value  using  contingency  tables.  The  lowest  R“  (U)  value  obtained  was  .0000  for 
Obligated  or  Deobligated  Dollars.  This  means  that  the  single  variable  model  with  the 
most  explanatory  power  is  the  model  that  includes  the  variable  Multiple  Award  Contract 
Fair  Opportunity  and  the  single  variable  model  with  the  least  explanatory  power  is  the 
model  that  includes  the  variable  Obligated  or  Deobligated  Dollars.  It  is  important  to  note 
that  some  of  the  variable  responses  were  classified  as  unstable,  meaning  that  a  regression 
model  could  not  be  established.  Variables  that  had  unstable  responses  included  Bundled 
Contract  Exception,  FSC  or  SVC  Code,  NAICS  Code,  Reason  Not  Synopsized, 
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Solicitation  Procedures,  Authority  for  Other  than  Full  and  Open  Competition,  and  Type 
of  Entity. 

In  an  attempt  to  build  a  model  for  prediction  of  PBS  A,  the  15  variables  classified 
as  stable  in  logistic  regression  were  cumulatively  tested  against  the  independent  variable 
PBSA.  When  using  all  15  stable  variables,  an  R2  (U)  of  .2019  was  obtained. 
Unfortunately,  many  of  the  parameter  estimates  for  the  variable  responses  in  the  model 
were  identified  as  being  biased.  This  indicates  that  there  are  one  or  more  linear 
dependencies  in  the  design  and  the  linear  combination  of  these  factors  exactly  duplicates 
another  factor's  value  (JMP,  2003).  These  variables  included  Type  of  Set-Aside,  Type  of 
Preference,  Women-Owned  Business,  Type  of  Contract,  Extent  Competed,  Synopsis,  and 
Contract/Order.  Consequently  all  of  these  variables  were  removed  from  the  model.  With 
both  the  unstable  and  biased  variables  removed  from  the  model  an  R”  (U)  of .  1615  was 
obtained.  Table  9  summarizes  these  results. 

Table  9.  Summary  of  Logistic  Regression  Model _ 

Variable  jp  Prob>  y2  R2  (U) 

Model  w/out  Unstable  or  Biased  Variables  15425.8  0.0000  0.1615 

This  model  includes  a  total  of  7  independent  variables.  The  parameter  estimates  for  the 
variable  responses  included  in  this  model  can  be  found  in  Appendix  H.  The  Wald 
statistic  y  value  was  used  to  test  the  significance  of  the  model.  The  y  value  and 
associated  P-value  support  the  rejection  of  the  null  hypothesis  that  the  parameter 
estimates  are  equal  to  zero. 

Separate  regressions  were  also  run  for  each  independent  variable  against  the 
dependent  variable,  PBSA.  Parameter  estimates  for  each  variable  response  were 
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calculated  using  logistic  regression  and  tested  using  the  null  and  alternate  hypotheses. 
The  probability  of  PBS  A  for  each  response  was  calculated  by  substituting  the  parameter 
estimate  and  intercept  values  into  the  logistic  regression  formula.  These  results  are 
displayed  in  Tables  10  through  25. 

Table  10.  Summary  of  Bundled  Contract  Response  Parameter  Estimates  and  Probability 
of  PBSA _ _ _ _ 


Response 

Estimate 

Standard 

Error 

2 

X 

Prob>x2 

Prob 

PBSA 

Intercept 

-0.1058595 

0.0729567 

2.11 

Bundled  Contract  [N] 

-0.784822 

0.0729567 

115.72 

0.2909 

Table  1 1 .  Summary  of  Obligated  or  Deobligated  Dollars  Parameter  Estimates  and 


Probability  of  PBSA 

Response 

Estimate 

Standard 

Error 

x2 

Prob>x2 

Prob 

PBSA 

Intercept 

-0.8857118 

0.0080113 

12223 

0.0000 

Obligated  or 
Deobligated  Dollars 

-7.626e-12 

8.454e-10 

0.00 

0.9928 

2.92E- 

01 

Table  12.  Summary  of  Contract/Order  Response  Parameter  Estimates  and  Probability  of 


PBSA 


Response 

Estimate 

Standard 

Error 

x2 

Prob>x2 

Prob 

PBSA 

Intercept 

-0.998353 

0.0358911 

773.74 

Contract/Order  [1] 

-0.7217959 

0.1237749 

34.01 

0.1518 

Contract/Order  [3] 

-0.2074712 

0.0380209 

29.78 

0.2304 

Contract/Order  [4] 

-0.9436227 

0.0952635 

98.12 

0.1254 

Contract/Order  [5] 

0.18104332 

0.0371747 

23.72 

0.3063 

Contract/Order  [6] 

0.79224635 

0.0405313 

382.07 

0.4486 

Contract/Order  [7] 

-0.0414182 

0.1740474 

0.06 

0.8119 

0.2611 

Contract/Order  [8] 

1.10775159 

0.0975163 

129.04 

0.5273 
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Table  13.  Summary  of  Type  of  Indefinite-Delivery  Contract  Response  Parameter 
Estimates  and  Probability  of  PBSA _ 


Response 

Estimate 

Standard 

Error 

x2 

Prob>x2 

Prob 

PBSA 

Intercept 

-1.0211204 

0.0219221 

2169.6 

0.0000 

Type  of  Indefinite- 
Delivery  Contract  [A] 

0.07728689 

0.0275839 

7.85 

0.0051 

0.2801 

Type  of  Indefinite- 
Delivery  Contract  [B] 

0.22509695 

0.023591 

91.04 

<.0001 

0.3108 

Type  of  Indefinite- 
Delivery  Contract  [C] 

-0.3855376 

0.0624557 

38.11 

<.0001 

0.1967 

Table  14.  Summary  of  Multiple  Award  Contract  Fair  Opportunity  Response  Parameter 

Estimates  and  Probability  of  PBSA 

Response 

Estimate 

Standard 

Error 

x2 

Prob>x2 

Prob 

PBSA 

Intercept 

-0.090934 

0.0655989 

1.92 

0.1657 

Multiple  Award  Contract 
Fair  Opportunity  [A] 

.1.474946 

0.0720001 

419.65 

<.0001 

0.1728 

Multiple  Award  Contract 
Fair  Opportunity  [B] 

-0.3685983 

0.3081339 

1.43 

<.0001 

0.3870 

Multiple  Award  Contract 
Fair  Opportunity  [C] 

-0.7424187 

0.0658268 

60.02 

<.0001 

0.3029 

Multiple  Award  Contract 
Fair  Opportunity  [D] 

0.33541685 

0.0849517 

15.59 

<.0001 

0.5608 

Multiple  Award  Contract 
Fair  Opportunity  [E] 

3.23880086 

0.0906054 

1277.8 

<.0001 

0.9588 

Table  15.  Summary  of  Indefinite-Delivery  Contract  Use  Response  Parameter  Estimates 
and  Probability  of  PBSA _ 


Response 

Estimate 

Standard 

Error 

x2 

Prob>x2 

Prob 

PBSA 

Intercept 

-0.6769128 

0.0327053 

428.38 

<.0001 

Indefinite -Delivery 
Contract  Use  [A] 

0.13168575 

0.0836275 

2.48 

0.1153 

0.3669 

Indefinite -Delivery 
Contract  Use  [B] 

-0.4508388 

0.0864576 

27.19 

<.0001 

0.2445 

Indefinite -Delivery 
Contract  Use  [C] 

0.06102411 

0.0671189 

0.83 

0.3632 

0.3507 

Indefinite -Delivery 
Contract  Use  [D] 

0.62824516 

0.036076 

303.26 

<.0001 

0.4878 
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Table  16.  Summary  of  Synopsis  Response  Parameter  Estimates  and  Probability  ofPBSA 


Response 

Estimate 

Standard 

Error 

r 

Prob>x2 

Prob 

PBSA 

Intercept 

-0.8755429 

0.0112263 

6080.5 

0.0000 

Synopsis  [A] 

0.0127636 

0.0133898 

0.91 

0.3405 

0.2967 

Synopsis  [B] 

-0.2978881 

0.0262107 

129.17 

<.0001 

0.2362 

Synopsis  [N] 

-0.3812092 

0.0168928 

509.24 

<.0001 

0.2215 

Table  17.  Summary  of  Extent  Competed  Response  Parameter  Estimates  and  Probability 
ofPBSA 

Response 

Estimate 

Standard 

Error 

r 

Prob>x2 

Prob 

PBSA 

Intercept 

-1.1004947 

0.0540626 

414.36 

Extent  Competed  [A] 

0.31589528 

0.0544904 

33.61 

Extent  Competed  [B] 

0.07087583 

0.0582072 

1.48 

Extent  Competed  [C] 

-0.4739783 

0.0757671 

39.13 

Extent  Competed  [D] 

-0.3875817 

0.0582598 

44.26 

Table  18.  Summary  of  Type  of  Contract  Response  Parameter  Estimates  and  Probability 
ofPBSA _ _  _ _ 


Response 

Estimate 

Standard 

Error 

r 

Prob>x2 

Prob 

PBSA 

Intercept 

-1.0594573 

0.0427791 

613.34 

Type  of  Contract  [A] 

-0.8376623 

0.4049809 

4.28 

0.0386 

0.1304 

Type  of  Contract  [J] 

0.3953269 

0.0440563 

80.52 

<.0001 

0.3398 

Type  of  Contract  [K] 

0.83916138 

0.0951713 

77.75 

<.0001 

0.4451 

Type  of  Contract  [L] 

-0.0684458 

0.0946795 

0.52 

0.4697 

0.2445 

Type  of  Contract  [M] 

0.91469634 

0.0821388 

124.01 

<.0001 

0.4638 

Type  of  Contract  [R] 

0.33138718 

0.0501084 

43.74 

<.0001 

0.3256 

Type  of  Contract  [S] 

-0.8601807 

0.0717832 

143.59 

<.0001 

0.1279 

Type  of  Contract  [T] 

-1.3640221 

0.2600419 

27.51 

<.0001 

0.0813 

Type  of  Contract  [U] 

-0.922947 

0.0485074 

362.02 

<.0001 

0.1210 

Type  of  Contract  [V] 

0.4370051 

0.0771864 

32.05 

<.0001 

0.3492 

Type  of  Contract  [Y] 

-0.1065667 

0.0497674 

4.59 

<.0001 

0.2375 

Type  of  Contract  [Z] 

0.39199958 

0.0804276 

23.76 

<.0001 

0.3390 
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Table  19.  Summary  of  Commercial  Items  Response  Parameter  Estimates  and  Probability 
ofPBSA _ _ _ _ 


Response 

Estimate 

Standard 

Error 

x2 

Prob>x2 

Prob 

PBSA 

Intercept 

-0.4863992 

0.0107112 

Commercial  Items  [N] 

-0.6438603 

0.0120437 

2858 

0.2441 

Commercial  Items  [Y] 

0.39160175 

0.016669 

551.91 

0.4763 

Table  20.  Summary  of  Women-Owned  Business  Response  Parameter  Estimates  and 

Probability  of  PBSA 

Response 

Estimate 

Standard 

Error 

x2 

Prob>x2 

Prob 

PBSA 

Intercept 

-0.714826 

0.0198109 

1301.9 

<.0001 

Women-Owned 
Business  [N] 

-0.2731451 

0.0207879 

172.65 

<.0001 

0.2713 

Women-Owned 
Business  [U] 

0.05253677 

0.0498592 

1.11 

0.2920 

0.3402 

Women-Owned 
Business  [Y] 

-0.2850085 

0.034124 

69.76 

<.0001 

0.2689 

Table  2 1 .  Summary  of  HUBZone  Representation  Response  Parameter  Estimates  and 
Probability  ofPBSA _ 


Response 

Estimate 

Standard 

Error 

x2 

Prob> 

x2 

Prob 

PBSA 

Intercept 

-1.111624 

0.0182761 

3699.6 

0.0000 

HUBZone  Representation  [N] 

-0.029883 

0.0203999 

2.15 

0.1430 

0.2420 

HUBZone  Representation  [Y] 

-0.317349 

0.0349925 

82.25 

<.0001 

0.1932 

Table  22.  Summary  of  Ethnic  Group  Response  Parameter  Estimates  and  Probability  of 


PBSA 


Response 

Estimate 

Standard 

Error 

x2 

Prob>x2 

Prob 

PBSA 

Intercept 

-1.1986828 

0.0246557 

2363.6 

Ethnic  Group  [A] 

-0.0632047 

0.0971424 

0.42 

0.5153 

0.2206 

Ethnic  Group  [B] 

0.31302413 

0.0597238 

27.47 

0.2920 

Ethnic  Group  [C] 

0.21764375 

0.0483839 

20.23 

0.2726 

Ethnic  Group  [D] 

-0.2523166 

0.0520292 

23.52 

0.1898 

Ethnic  Group  [E] 

-0.1670177 

0.045031 

13.76 

0.2033 

Ethnic  Group  [F] 

-0.3618537 

0.0961776 

14.16 

0.1735 

Ethnic  Group  [Z] 

-0.0550928 

0.0636474 

0.75 

0.3867 

0.2220 
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Table  23.  Summary  of  Tvpe  of  Set-Aside  Response  Parameter  Estimates  and  Probability 
ofPBSA _ _  _ _ 


Response 

Estimate 

Standard 

Error 

x2 

Prob>x2 

Prob 

PBSA 

Intercept 

-1.1581767 

0.050894 

517.87 

<.0001 

Type  of  Set-Aside  [A] 

0.19917358 

0.0516292 

14.88 

0.0001 

0.2770 

Type  of  Set-Aside  [B] 

0.36304191 

0.0553928 

42.95 

<.0001 

0.3110 

Type  of  Set-Aside  [C] 

-0.210333 

0.1277981 

2.71 

0.0998 

0.2028 

Type  of  Set-Aside  [D] 

-0.1492011 

0.0564274 

6.99 

0.0082 

0.2129 

Type  of  Set-Aside  [E] 

-0.1824305 

0.1418217 

1.65 

0.1983 

0.2074 

Type  of  Set-Aside  [F] 

-0.6467775 

0.2793333 

5.36 

0.0206 

0.1412 

Type  of  Set-Aside  [K] 

-0.0391031 

0.1176526 

0.11 

0.7396 

0.2319 

Type  of  Set-Aside  [L] 

-0.2833379 

0.2151878 

1.73 

0.1879 

0.1913 

Table  24.  Summary  of  Type  of  Preference  Response  Parameter  Estimates  and  Probability 
ofPBSA _ _ _ _ 


Response 

Estimate 

Standard 

Error 

x2 

Prob>x2 

Prob 

PBSA 

Intercept 

-0.9750136 

0.3291877 

8.77 

0.0031 

Type  of  Preference  [A] 

-0.0079431 

0.329244 

0.00 

0.9808 

0.2723 

Type  of  Preference  [B] 

-0.1235974 

0.8803583 

0.02 

0.8883 

0.2500 

Type  of  Preference  [D] 

-0.634264 

0.5552866 

1.30 

0.2534 

0.1666 

Table  25.  Summary  of  SBIR  Program  Response  Parameter  Estimates  and  Probability  of 


PBSA 


Response 

Estimate 

Standard  Error 

x2 

Prob>x2 

Prob 

PBSA 

Intercept 

-1.3166613 

0.0930408 

200.26 

<.0001 

SBIR  Program  [A] 

0.35851964 

0.0932893 

14.77 

0.0001 

0.2772 

SBIR  Program  [B] 

-0.4535634 

0.3104716 

2.13 

0.1440 

0.1455 

SBIR  Program  [C] 

-0.3433492 

0.1012258 

11.51 

0.0007 

0.1597 

SBIR  Program  [D] 

.0.6444348 

0.2210532 

8.50 

0.0036 

0.1233 

The  null  hypothesis  is  rejected  for  all  responses  with  P-values  <  .01.  This  means 
that  the  selection  of  these  particular  responses  do  not  change  the  probability  of  an  action 
being  PBSA.  For  all  other  responses,  where  the  null  hypothesis  was  rejected,  a  positive 
parameter  estimate  value  signifies  an  increase  in  the  probability  of  an  action  being  PBSA 
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when  that  particular  response  is  chosen.  A  negative  value  signifies  a  decrease  in  the 
probability  of  an  action  being  PBSA.  A  majority  of  the  parameter  estimates  were 
negative.  When  a  response  parameter  estimate  added  with  its  corresponding  intercept 
parameter  estimate,  resulted  in  a  positive  value,  the  probability  of  that  action  being  PBSA 
was  greater  than  .5.  Conversely,  when  a  response  parameter  estimate  added  with  its 
corresponding  intercept  parameter  estimate,  resulted  in  a  negative  value,  the  probability 
of  that  action  being  PBSA  was  less  than  .5. 

By  far,  the  most  positively  influential  response  variable  is  Multiple  Award 
Contract  Fair  Opportunity  response  “E”  with  a  parameter  estimate  of  3.2388  and  a 
probability  of  PBSA  of  .9588.  This  means  when  response  “E”  is  selected,  the  probability 
of  that  action  being  PBSA  is  almost  96%.  Response  “E”  is  Minimum  Guarantee  and  is 
coded  when  it  was  necessary  to  place  an  order  to  satisfy  a  minimum  amount  guaranteed 
to  the  contractor.  The  next  most  positively  influential  response  variable  is 
Contract/Order  response  “8”  with  a  parameter  estimate  of  1.1078  and  probability  of 
PBSA  of  .5273.  This  means  when  response  “8”  is  selected,  the  probability  of  that  action 
being  PBSA  is  over  52%.  Response  “8”  is  Order  from  Procurement  List  and  is  entered 
when  the  contracting  action  is  an  action  placed  with  Federal  Prison  Industries  (UNICOR) 
or  a  JWOD  Participating  Nonprofit  Agency. 

The  most  negatively  influential  response  variable  is  Type  of  Contract  response 
“T,”  with  parameter  estimate  of -1.36402  and  a  probability  of  PBSA  of  .0813.  This 
means  when  response  “T”  is  selected,  the  probability  of  that  action  being  PBSA  is  8.13%. 
Response  “T”  is  the  designation  for  a  Cost-Sharing  contract  The  next  most  negatively 
influential  response  variable  is  Multiple  Award  Contract  Fair  Opportunity  response  “A” 
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with  a  parameter  estimate  of -1.4749  and  a  probability  of  PBSA  of  .17228.  This  means 
when  response  “A”  is  selected,  the  probability  of  that  action  being  PBSA  is  17.23%. 
Response  “A”  is  Fair  Opportunity  Process  and  is  entered  when  the  delivery  or  task  order 
was  issued  pursuant  to  a  process  that  permitted  each  contract  awardee  a  fair  opportunity 
to  be  considered. 

Summary 

This  chapter  presented  and  discussed  the  research  results.  A  sample  of  PBSA 
contracts  taken  from  three  contracting  offices  were  found  to  be  consistent  with  the 
minimum  criteria  identified  in  the  PBSA  Guidebook,  suggesting  that  there  may  not  be  a 
systematic  problem  with  PBSA  coding.  Using  descriptive  statistics,  it  was  concluded  that 
the  USAF  is  not  meeting  interim  PBSA  goals  and  the  percentage  of  modification  dollars 
coded  PBSA  is  not  equal  to  the  percentage  of  non-modification  dollars  coded  PBSA. 
Additionally,  forecasting  suggested  that  the  USAF  will  not  meet  the  FY  2005  goal  of 
50%  of  all  actions  and  dollars  PBSA.  Lastly,  contingency  tables  and  logistic  regression 
identified  dependent  variables  and  variable  responses  that  increase  and  decrease  the 
probability  of  an  action  being  PBSA.  Chapter  5  will  discuss  the  conclusions  and 
recommendations  stemming  from  these  results,  address  the  limitations  of  the  study,  and 
make  suggestions  for  future  research. 
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V.  Conclusions  and  Recommendations 


Introduction 

The  purpose  of  this  study  was  to  evaluate  and  analyze  the  current  and  expected 
future  states  of  Performance  Based  Service  Acquisition  (PBS  A)  implementation  in  the 
United  States  Air  Force  (USAF)  in  order  to  assist  USAF  leadership  in  making  PBSA 
implementation  decisions.  This  chapter  outlines  the  conclusions  drawn  from  the  results 
of  this  study  and  offers  recommendations  for  PBSA  implementation  improvements.  Also 
included  in  this  chapter  are  the  limitations  of  the  study  and  suggestions  for  future 
research. 

Conclusions  and  Recommendations 

A  total  of  five  investigative  questions  were  established  in  order  to  answer  and 
address  the  over  arching  research  question  of,  “What  are  the  current  and  expected  future 
states  of  PBSA  implementation  in  the  USAF?” 

Conclusions  -  Investigative  Question  1.  Is  the  J001  database,  which  is  the  source 
of  government  procurement  data,  reliable?  The  answer  to  this  question  is  still  unknown. 
However,  this  research  concludes  that  the  coding  of  PBSA  contracts  for  the  88  ABW/PK, 
AFRL  Det  1/PK,  and  ASC/PK  is  reliable  and  consistent  with  the  minimum  mandatory 
criteria  identified  in  The  Guidebook  for  Performance-Based  Services  Acquisitions  in  the 
Department  of  Defense  (PBSA  Guidebook).  While  this  determination  of  reliability 
cannot  be  generalized  statistically  to  the  entire  J001  database,  it  does  indicate  that  there 
may  not  be  systematic  problems  with  PBSA  coding.  For  PBSA  coding,  these  results  are 
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contrary  to  past  GAO  findings  that  suggest  federal  procurement  data  is  unreliable 
(Woods,  2003).  Additionally,  the  results  imply  that  the  J001  database  may  not  be  as 
inaccurate  as  initially  reported. 

Recommendations  -  Investigative  Question  1.  Because  the  results  suggest  that 
there  may  not  be  a  systematic  problem  with  PBSA  coding  in  the  J001  database,  it  is 
recommended  that  the  USAF  increase  utilization  of  the  J001  database  for  PBSA 
performance  tracking.  Many  organizations  rely  on  self-reports  from  other  agencies  when 
compiling  PBSA  metrics.  From  the  researcher’s  personal  experience,  these  self-reports, 
many  times,  differ  from  what  was  actually  coded  on  the  DD  Form  350.  In  addition,  this 
self-reports  take  time  and  money  to  generate.  Utilizing  the  J001  database  may  ultimately 
be  the  most  accurate  and  efficient  means  of  measuring  PBSA  perfonnance  and  should  be 
used  to  the  maximum  extent  practicable. 

Conclusions  -  Investigative  Question  2.  Is  the  USAF  meeting  interim  PBSA 
goals?  The  answer  to  this  question  is  no.  The  USAF  was  on  track  and  had  met  all 
interim  PBSA  goals  established  by  the  acting  Under  Secretary  of  Defense  for 
Acquisition,  Technology  and  Logistics,  Mr.  Michael  Wynne,  through  FY  2003.  This  was 
large  in  part  due  to  the  fact  that  goals  were  set  low  and  were  easily  attainable.  However, 
in  FY  2004  the  percentage  of  dollars  awarded  using  PBSA  significantly  dropped  and  was 
well  below  the  FY  2004  goal  of  35  percent. 

The  decrease  in  PBSA  percentages  implies  several  possible  alternatives.  First,  the 
decrease  implies  that  the  emphasis  on  PBSA  in  waning.  Now  in  its  fourth  year,  PBSA 
implementation  may  not  be  as  much  of  as  hot  topic  as  it  was  upon  initial  start-up. 
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Conversely,  the  decrease  in  PBSA  percentages  may  imply  that  there  has  been  an 
increase  in  education,  training,  and  awareness  of  PBSA  techniques  and  application  and 
contracting  personnel  are  better  at  determining  which  contracts  should  use  PBSA.  Of  the 
3  incorrectly  coded  PBSA  contracts  sampled  in  this  study,  2  of  them  were  awarded 
during  the  first  2  years  of  PBSA  implementation.  This  suggests  that  PBSA  coding  may 
be  becoming  more  accurate  as  years  progress.  As  PBSA  coding  becomes  more  accurate, 
fewer  incorrectly  coded  contracts  may  be  contributing  to  PBSA  percentages.  The 
decrease  in  PBSA  percentages  during  FY  2004  may  not  be  as  significant  due  to  the  fact 
that  the  percentages  during  the  previous  years  may  have  been  over  inflated  with 
incorrectly  coded  PBSA  contracts. 

Lastly,  the  decrease  in  PBSA  percentages  may  imply  that  the  USAF  has  hit  a 
natural  plateau  in  PBSA.  The  literature  shows  no  rationale  for  establishment  of  the 
interim  PBSA  goals,  other  than  progression  towards  the  FY  2005  goal  of  50%.  The 
commercial  sector  emphasizes  PBSA  very  little  due  to  the  small  percentage  of  dollars 
attributed  to  the  procurement  of  services  (Barry,  2003).  Maybe  it  is  impractical  to  expect 
PBSA  percentages  greater  than  35%.  The  results  show  that  achieving  20%  to  30%  PBSA 
for  both  actions  and  dollars  is  possible.  However,  at  what  point  are  services  that  may  not 
be  suitable  for  PBSA  being  forced  into  PBSA  methods  for  the  sake  of  meeting  mandated 
goals?  Because  of  this,  the  question  that  surfaces  is,  “Were  these  goals  reasonable  and 
attainable  to  begin  with?”  This  research  suggests  that  they  weren’t. 

Recommendations  -  Investigative  Question  2.  It  is  recommended  that  the  USAF 
leadership  reinvigorate  the  emphasis  on  the  use  of  PBSA.  This  can  be  done  by  simply 
increasing  PBSA  communication  up  and  down  the  chain  of  command.  While  there  are 
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endless  methods  to  communicate  the  emphasis,  leadership  could  significantly  increase 
organizational  awareness  of  PBSA  by  continually  posting  current  PBSA  percentages  and 
PBSA  percentage  goals  for  each  organization  or  agency. 

It  is  also  recommended  that  USAF  leadership  revaluate  PBSA  goals.  Instead  of 
establishing  blanket  percentage  goals  for  all  services  actions,  percentages  should  be 
established  only  for  contract  characteristics  representing  a  large  percentage  of  the  total 
dollars  or  actions  and  a  high  probability  of  being  PBSA.  For  instance,  it  may  not  be  as 
important  to  reach  a  goal  of  50%  of  all  PBSA  dollars  for  Cost  Plus  Incentive  Fee 
contracts  when  these  type  of  contracts  accounted  for  less  than  2%  of  the  total  dollars  for 
FY  2004  and  are  seldom  used  with  PBSA.  However,  it  may  be  important  to  focus  on  the 
percentage  of  PBSA  dollars  for  Finn  Fixed  Price  contracts  since  they  accounted  for  the 
largest  percentage  of  total  dollars  for  FY  2004,  at  over  36%,  and,  when  compared  to  other 
responses,  have  a  relatively  high  probability  of  PBSA  at  33%.  This  will  lead  to  focused 
PBSA  efforts  directed  at  areas  that  make  the  most  sense  for  PBSA  use  and  are  the  most 
fiscally  influential.  While  this  may  not  lead  to  the  achievement  of  50%  of  all  actions  and 
dollars  PBSA,  it  can  lead  to  the  maximizing  of  PBSA  efforts  in  the  areas  that  represent 
the  largest  percentage  of  total  dollars. 

Conclusions  -  Investigative  Question  3.  Is  the  USAF  on  track  to  meet  future 
mandated  PBSA  goals?  The  answer  to  this  question  is  no.  Even  though  there  were  only 
four  data  points,  several  forecasting  methods  were  used.  Using  liner  regression  and 
moving  averages,  forecasts  of  the  percentage  actions  and  dollars  PBSA  for  FY  2005  were 
calculated.  All  forecasts  indicate  that  the  USAF  will  fall  well  short  of  the  50%  PBSA 
goal  for  FY  2005.  Although  the  forecasting  models  used  were  questionable,  the 
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graphical  evidence  illustrates  a  trend  line  falling  well  short  of  50%  for  FY  2005.  There 
was  clearly  an  upward  trend  in  the  percentages  of  PBSA  prior  to  FY  2004,  possibly 
meaning  that  the  decrease  in  FY  2004  was  an  anomaly  and  an  upward  trend  may  resume 
for  FY  2005.  However,  in  order  to  make  a  truly  reliable  forecast  more  data  points  would 
need  to  be  included  in  the  models. 

These  results  support  Lacey’s  (2004)  suspicion  that  the  USAF  will  not  meet  the 
FY  2005  goal  of  50%  PBSA.  Similarly  to  the  conclusions  for  investigative  question  2, 
the  results  imply  that  the  goal  may  not  have  been  reasonable  or  attainable  to  begin  with 
and  that  the  USAF  may  have  hit  a  natural  plateau  in  PBSA. 

Recommendations  -  Investigative  Question  3.  Similar  to  investigative  question  1, 
it  is  recommended  that  USAF  leadership  reinvigorate  the  emphasis  on  PBSA  use  and 
revaluate  PBSA  goals. 

Conclusions  -  Investigative  Question  4.  Is  the  percentage  of  modifications  coded 
PBSA  equal  to  the  percentage  of  non-modifications  coded  PBSA?  The  answer  to  this 
question  is  no.  While  the  number  of  actions  is  relatively  equal,  the  percentage  of  dollars 
PBSA  for  modifications  is  significantly  smaller  than  the  percentage  of  dollars  PBSA  for 
non-modifications.  Additionally,  the  cost  per  PBSA  modification  has  decreased  each 
year  and  been  less  than  the  cost  per  non-PBSA  modifications  every  year.  Conversely,  the 
cost  per  PBSA  non-modification  has  been  greater  than  the  cost  per  non-PBSA  non¬ 
modification.  These  results  imply  that  new  PBSA  contracts,  on  average,  cost  more  than 
non-PBSA  contracts,  but  result  in  modifications  that,  on  average,  cost  less  than  non- 
PBSA  modifications. 
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Recommendations  -  Investigative  Question  4.  It  is  recommended  that,  when 
deciding  whether  to  award  a  contract  using  PBSA  techniques,  contracting  personnel  take 
into  consideration  that  future  modifications  to  PBSA  contracts  may  cost  less  than  non- 
PBSA  contracts.  With  less  money  being  spent  on  modifications  to  PBSA  contracts,  a 
large  percentage  of  future  funds  can  be  used  to  fund  the  procurement  of  new  services  or 
supplies. 

Conclusions  -  Investigative  Question  5.  Are  any  particular  acquisition 
characteristics  associated  with  PBSA  use?  The  answer  to  this  question  is  yes. 
Contingency  tables  clearly  indicated  that  there  are  numerous  contract  characteristics  that 
are  dependent  with  PBSA,  confirming  Lacey’s  (2004)  ascertains.  However,  contingency 
tables  are  unable  to  detennine  the  extent  of  the  dependencies. 

Logistic  Regression  was  able  to  identify  which  variable  responses  are  significant 
and  whether  they  increase  or  decrease  the  probability  of  an  action  being  PBSA. 

Numerous  variables  were  found  to  be  significant,  but  very  few  of  the  variable  responses 
increased  the  probability  of  an  action  being  PBSA  above  50  percent. 

These  results  imply  that  PBSA  is  a  complex  type  of  contract  that  can  be  used  in 
various  types  of  contracting  circumstances.  Predicting  when  to  use  PBSA  can  not  easily 
be  done  by  knowing  one,  two,  or  even  “x”  corresponding  variables.  The  use  of  PBSA 
should  not  be  eliminated  as  a  possibility  simply  because  a  certain  variable  response  is 
selected.  While  some  responses  may  increase,  such  as  Multiple  Award  Contract  Fair 
Opportunity  response  “E,”  or  decrease  the  probability  of  PBSA,  such  as  Type  of  Contract 
response  “T,”  none  guarantee  or  completely  eliminate  the  possibility  of  PBSA  use. 


95 


Recommendations  -  Investigative  Question  5.  It  is  recommended  that  contracting 
personnel  look  at  all  variables  surrounding  an  acquisition  when  determining  whether  to 
use  PBSA  techniques.  The  use  of  PBSA  should  not  be  dependent  on  the  selection  of  a 
single  variable  response.  The  results  of  this  study  show  that  there  are  numerous 
dependencies  and  correlations  between  contract  characteristics  and  PBSA.  PBSA 
determination  based  on  a  single  response  should  be  avoided. 

Study  Limitations 

Just  like  any  research,  there  are  some  limitations  to  this  study.  Factors  that 
contributed  to  these  shortcomings  included  funding  limitations,  time  constraints,  and 
practicality. 

The  first  limitation  to  this  research  is  the  reliability  of  the  J001  database. 

Although  a  sample  of  PBSA  contracts  was  taken  and  evaluated  against  the  minimum 
mandatory  PBSA  criteria,  it  did  not  represent  PBSA  contracts  throughout  the  USAF,  only 
the  three  offices  from  which  it  was  taken.  Therefore,  the  reliability  of  the  J001  database 
across  the  USAF  is  still  unknown.  An  effort  was  made  to  select  different  types  of 
contracting  offices,  which  work  with  different  types  of  services  contracts.  However,  the 
sample  was  unable  to  account  for  every  demographic  in  USAF  contracting.  Additionally, 
while  the  results  yielded  from  this  sample  were  satisfactory,  the  results  from  a  sample 
taken  from  the  entire  USAF  may  significantly  differ,  thus  yielding  a  contradictory 
determination  of  the  reliability  of  the  J001  database. 

The  second  limitation  of  this  study  is  the  fact  that  when  evaluating  the  sampled 
contracts  against  the  minimum  mandatory  PBSA  criteria,  the  quality  of  the  PBSA  service 
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contracts  was  not  taken  into  account.  For  instance,  a  PBSA  service  contract  that 
contained  well  thought-out,  specific,  and  easily  measured  performance  evaluation 
measurements  was  considered  the  same  as  a  contract  that  met  the  bare  minimum 
requirements  for  PBSA  classification.  Because  of  this,  it  is  not  known  if  better  quality 
PBSA  contracts  differ  from  minimum  quality  PBSA  contracts,  and  thus  yield  different 
results. 

Another  limitation  of  the  study  is  the  use  of  forecasting  methods  to  forecast  FY 
2005  percentages.  Because  there  was  only  four  years  of  data  to  build  a  predictive  model, 
the  linear  regression  and  2-period  moving  average  models  established  and  used  to 
forecast  were  not  optimal.  Significance  of  the  models  was  limited,  but  they  did  provide 
an  illustration  of  the  general  trend  of  the  data. 

Future  Research 

With  so  little  research  conducted  in  this  field  and  still  so  many  questions  to  be 
answered,  the  prospects  for  future  research  are  plentiful.  Some  of  these  research  ideas 
stem  from  questions  that  surfaced  from  the  results  of  this  study,  while  some  are  simply 
questions  that  have  yet  to  be  answered. 

The  first  opportunity  for  future  research  is  to  confirm  this  study’s  results  that  the 
J001  database  is  truly  reliable,  not  only  for  the  PBSA  contracts  at  the  three  contracting 
offices  sampled  at  Wright-Patterson  AFB,  OH,  but  for  all  contracts  across  the  entire 
USAF.  This  could  be  done  by  taking  a  sample  of  PBSA  coded  contracts  from  the  entire 
pool  of  PBSA  contracts  and  evaluating  them  against  the  minimum  mandatory  PBSA 
criteria,  similarly  to  the  way  in  which  investigative  question  1  in  this  study  was  answered. 
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Careful  stratification  of  the  sample  would  need  to  be  taken  in  order  to  ensure  all  USAF 
demographics  were  properly  represented.  Not  until  this  type  of  sample  is  evaluated  and 
tested,  will  USAF  leadership  truly  know  the  reliability  of  PBSA  coding  in  the  J001 
database. 

The  next  opportunity  for  future  research  would  be  to  further  the  logistic 
regression  analysis  performed  in  this  study  by  building  a  comprehensive  model  for 
prediction  of  PBSA  using  contracts  variable  responses.  It  was  apparent  during  this  study 
that  there  are  numerous  variables  that  are  dependent  with  PBSA.  Using  this  study’s 
results  as  a  foundation,  the  research  could  solidify  which  variable  responses,  if  any,  when 
combined,  significantly  increase  the  probability  of  a  contracting  action  being  PBSA. 

This  could  be  done  by  using  other  functional  forms  or  combinations  of  variables.  The 
establishment  of  a  predictive  model  may  lead  to  underlying  relationships  with  PBSA  that 
are  transparent  on  the  surface.  However,  any  model  would  have  to  be  evaluated  for 
practical  significance  in  order  to  provide  any  benefit  to  contracting  personnel. 

The  next  possible  research  effort  would  be  to  evaluate  what  is  causing  the 
fluctuation  in  PBSA  percentages  for  different  contract  types  from  year  to  year.  The 
results  from  this  study  clearly  show  significant  changes  in  PBSA  percentages  for 
particular  contract  variables,  such  as  Type  of  Contract,  from  year  to  year.  Why  these 
changes  are  occurring  and  what  is  causing  them  is  unknown.  Some  of  these  fluctuations 
may  be  insignificant  because  of  the  small  percentage  of  the  total  dollars  or  actions  the 
response  may  represent,  and  not  be  worth  investigation.  Investigating  the  fluctuation  of 
variable  responses  that  account  for  a  large  percentage  of  the  total  dollars  may  reveal  the 
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true  cause  of  the  decline  in  PBSA  percentages  in  FY  2004  and  lead  to  the  true  driver  of 
PBSA  use. 

The  next  opportunity  for  future  research  would  be  to  study  the  relationship 
between  service  codes  and  PBSA  use.  By  identifying  which  individual  service  codes 
increase  and  decrease  the  probability  of  PBSA  and  account  for  the  largest  percentage  of 
total  dollars  and/or  actions,  USAF  leadership  could  gain  a  better  understanding  of  what 
services  do  and  do  not  facilitate  the  use  of  PBSA  techniques.  Using  this  infonnation, 
focused  PBSA  goals  for  individual  service  codes  could  be  established. 

Another  possible  area  of  study  would  be  the  evaluation  of  cost  savings  associated 
with  use  of  PBSA.  Of  course,  studies  (OFPP,  1998)  have  concluded  there  are  benefits 
are  associated  with  PBSA  use.  However,  no  recent  studies  have  shown  how  much  the 
USAF  is  saving,  if  any,  by  implementing  and  using  PBSA.  If  the  USAF  is  not  saving  any 
money  or  gaining  any  quality  by  implementing  and  using  PBSA,  why  spend  the  time  and 
resources  on  PBSA  training  and  the  achievement  of  PBSA  goals?  Additionally,  if  the 
USAF  is  achieving  cost  savings  from  PBSA,  how  does  quality  of  the  PBSA  contracts 
affect  the  savings?  This  study  did  not  account  for  quality  differences  between  PBSA 
contracts,  as  all  PBSA  contracts  were  deemed  to  be  equal.  It  would  be  interesting  to 
know  if  better  quality  PBSA  contracts  yield  more  savings  and/or  better  results  than  PBSA 
contracts  that  only  meet  the  minimum  PBSA  criteria. 

Conclusion 

The  USAF  has  made  tremendous  strides  in  the  implementation  of  PBSA. 
However,  it  is  still  falling  short  of  interim  goals  and  will  most  likely  fall  short  of  the 
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mandated  FY  2005  goal.  However,  the  question  of  whether  these  goals  were  reasonable 
or  attainable  still  remains  unanswered.  Since  the  USAF  has  no  equal  in  the  commercial 
sector  for  comparison,  it  is  impossible  to  know  if  such  goals  are  reachable  or  if  the 
USAF  has  hit  a  natural  plateau  in  PBSA  use  never  experienced  by  any  other  organization. 
Instead  of  trying  to  convert  everything  to  PBSA,  USAF  leadership  should  focus  on 
maximizing  PBSA  in  areas  that  are  practical  and  fiscally  influential  to  the  overall 
percentage  of  total  dollars,  such  as  Finn  Fixed  Price  type  contracts.  While  this  may 
result  in  an  overall  percentage  of  PBSA  less  than  50%,  it  will  lead  to  more  efficient  use 
of  personnel  and  resources. 
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Appendix  A:  Service  Code  Categories 


B.  Special  Studies  &  Analysis  -  Not  R&D 

C.  Architect  &  Engineering  Services 

D.  Automatic  Data  Processing  &  Telecommunication  Services 

E.  Purchase  of  Structures  or  Facilities 

F.  Natural  Resources  &  Conservation  Services 

G.  Social  Services 

H.  Quality  Control,  Testing  &  Inspection  Services 

J.  Maintenance,  Repair  &  Rebuilding  of  Equipment 

K.  Modification  of  Equipment 

L.  Technical  Representative  Services 

M.  Operation  of  Government-Owned  Facilities 

N.  Installation  of  Equipment 

P.  Salvage  Services 

Q.  Medical  Services 

R.  Professional,  Administrative  &  Management  Support  Services 

S.  Utilities  &  Housekeeping  Services 

T.  Photographic,  Mapping,  Printing  &  Publication  Services 

U.  Educational  &  Training  Services 

V.  Transportation,  Travel  &  Relocation  Services 

W.  Lease  or  Rental  of  Equipment 

X.  Lease  or  Rental  of  Facilities 

Y.  Construction  of  Structures  &  Facilities 

Z.  Maintenance,  Repair  or  Alteration  of  Real  Property 
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Appendix  B:  Summary  of  Percentage  of  Actions  and  Dollars  PBSA  by  Fiscal  Year 


FY  2001 

Total 

PBSA 

%  PBSA 

Dollars 

$19,733,040,849 

$5,038,331,141 

25.53% 

Actions 

15714 

3239 

20.61% 

FY  2002 

Total 

PBSA 

%  PBSA 

Dollars 

$22,310,837,208 

$6,471,973,020 

29.01% 

Actions 

18500 

5162 

27.90% 

FY  2003 

Total 

PBSA 

%  PBSA 

Dollars 

$28,511,634,144 

$9,199,368,959 

32.27% 

Actions 

21943 

7401 

33.73% 

FY  2004 

Total 

PBSA 

%  PBSA 

Dollars 

$26,649,991,736 

$7,663,018,554 

28.75% 

Actions 

20581 

6605 

32.09% 
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Appendix  C:  Summary  of  Percentage  of  Actions  and  Dollars  PBSA  for  Service 
Category,  Type  of  Contract,  and  Type  of  Entity  for  Fiscal  Year  2001 


Total 


Dollars 


$8,949,600,040 


$323,351,090 


$810,075,362 


$0 


$43,189,084 


$22,675,781 


$45,715,319 


$0 


$1,599,715,391 


$861,485,895 


$655,351,291 


$595,101,263 


$25,612,730 


$0 


$21,974,678 


$14,457,567 


$2,883,200,951 


$628,812,172 


$50,781,733 


$169,087,643 


$809,544,498 


$72,957,987 


$10,870,353 


$1,139,480,021 


$19,733,040,849 


ESEiai 


FY  2001 


PBSA 


Dollars 


$1,472,290,609 


$39,381,294 


$251,945,817 


$0 


$7,234,215 


$127,072 


$1,540,286 


$0 


$227,989,005 


$67,917,913 


$607,736,150 


$322,396,915 


$7,022,123 


$0 


$3,381,878 


$2,426,559 


$933,254,934 


$312,262,807 


$9,435,215 


$45,850,191 


$708,031,511 


$15,664,421 


$2,442,226 


$0 


$5,038,331,141 


EBjEUi 


%  PBSA 


Dollars  Actions 


16.45% 


12.18% 


31.10% 


0.00% 


16.75% 


0.56% 


3.37% 


0.00% 


14.25% 


7.88% 


92.73% 


54.18% 


27.42% 


0.00% 


15.39% 


16.78% 


32.37% 


49.66% 


18.58% 


27.12% 


87.46% 


21.47% 


22.47% 


0.00% 


25.53% 


5.82% 


14.08% 


34.03% 


0.00% 


32.10% 


4.55% 


18.52% 


0.00% 


15.51% 


3.77% 


66.28% 


54.66% 


31.25% 


0.00% 


17.02% 


29.03% 


32.55% 


50.55% 


23.46% 


19.42% 


66.24% 


31.15% 


40.00% 


0.00% 


20.61% 


%  Dollars  of  Total 


45.35% 


1 .64% 


4.11% 


0.00% 


0.22% 


0.11% 


0.23% 


0.00% 


4.37% 


3.32% 


3.02% 


0.13% 


0.00% 


0.11% 


0.07% 


14.61% 


3.19% 


0.26% 


0.86% 


4.10% 


0.37% 


0.06% 


5.77% 


100.00% 


C5  -  Type  of  Contract 


Blank 


A  -  Firm  Fixed  Price  Redeterm 


J  -  Firm  Fixed  Price 


K  -  Fixed  Price  Econ  Price  Adj 


L  -  Fixed  Price  Incentive 


M  -  Fixed  Price  Award  Fee 


R  -  Cost  Plus  Award  Fee 


S  -  Cost  Contract 


T  -  Cost  Sharing 


U  -  Cost  Plus  Fixed  Fee 


V  -  Cost  Plus  Incentive  Fee 


Y  -  Time  and  Materials 


Z  -  Labor  Hour 


Total 


Total 


Dollars 


$911,955,691 


$132,953,058 


$7,121,978,511 


$221,571,764 


$545,378,522 


$168,635,400 


$6,382,828,278 


$613,186,136 


$21,574,975 


$1,916,305,108 


$379,635,498 


$1,135,150,468 


$181,887,440 


$19,733,040,849 


KWMlKlI 


FY  2001 


PBSA 


Dollars 


$341,675,527 


$1,099,964 


$2,342,189,011 


$45,583,739 


$71,840,788 


$83,696,954 


$1,775,037,365 


$3,015,009 


$575,084 


$175,533,215 


$8,770,228 


$109,080,298 


$80,233,959 


$5,038,331,141 


ESSE  I 


%  PBSA 


Dollars  Actions 


37.47% 


0.83% 


32.89% 


20.57% 


13.17% 


49.63% 


27.81% 


0.49% 


2.67% 


9.16% 


2.31% 


9.61% 


44.11% 


25.53% 


37.56% 


8.70% 


24.25% 


39.10% 


13.14% 


33.86% 


23.92% 


2.75% 


6.00% 


6.43% 


8.33% 


14.11% 


42.49% 


20.61% 


%  Dollars  by  Total 


4.62% 


0.67% 


36.09% 


1.12% 


2.76% 


0.85% 


32.35% 


3.11% 


0.11% 


9.71% 


1 .92% 


5.75% 


0.92% 


100.00% 


D1A  -  Type  of  Entity 


Blank 


A  -  Small  Disadvantaged  in  U.S. 


B  -  Other  Small  in  U.S. 


C  -  Large  in  U.S. 


D  -  JWOD  Nonprofit  Agency 


F  -  Hospital 


L  -  Foreign  Concern/Entity 


M  -  Domestic  Firm  Outside  U.S. 


T  -  Historically  Black  College 


U  -  Minority  Institution 


V  -  Other  Educational 


Z  -  Other  Nonprofit 


Total 


Total  j 

Dollars 

$911,955,691 

HSUS! 

$1,269,886,249 

2314  1 

$1,635,130,149 

$13,751,598,800 

7404  | 

$121,546,408 


$575,590 


$433,639,071 


$384,562,295 


$5,054,257 


$167,561,977 


$1,050,632,006 


$19,733,040,849 


FY  2001 


PBSA 


Dollars 


$341,675,527 


$350,000,693 


$239,884,021 


$3,448,422,796 


$38,409,844 


$575,590 


$100,734,073 


$113,036,133 


$0 


$447,997 


$3,143,090 


$402,001,377 


$5,038,331,141 


%  PBSA 


Dollars  Actions 


37.47% 


27.56% 


14.67% 


25.08% 


31 .60% 


37.56% 


19.58% 


14.16% 


19.30% 


37.43% 


23.23% 


29.39% 


0.00% 


8.86% 


1 .88% 


38.26% 


25.53% 


22.84% 


50.83% 


0.00% 


12.50% 


2.90% 


15.23% 


20.61% 


%  Dollars  by  Total 


4.62% 


6.44% 


8.29% 


69.69% 


0.62% 


0.00% 


2.20% 


1 .95% 


0.00% 


0.03% 


0.85% 


5.32% 


100.00% 


Appendix  D:  Summary  of  Percentage  of  Actions  and  Dollars  PBSA  for  Service 
Category,  Type  of  Contract,  and  Type  of  Entity  for  Fiscal  Year  2002 


Total 


Dollars 


$9,384,582,101 


$493,684,925 


$863,411,994 


$9,069,698 


$84,659,024 


$15,761,156 


$49,354,140 


$0 


$1,842,423,172 


$408,947,473 


$901,123,782 


$682,583,799 


$34,621,212 


$0 


$19,886,640 


$17,871,097 


$3,561,532,312 


$856,956,221 


$43,995,336 


$236,157,686 


$1,569,942,101 


$83,375,503 


$10,902,510 


$1,139,995,326 


$22,310,837,208 


ESEOSl 


■EEfflill 


FY  2002 


PBSA 


Dollars 


$911,685,245 


$65,451,003 


$404,155,906 


$3,030,064 


$18,042,034 


$543,637 


$5,992,026 


$0 


$454,181,233 


$46,903,111 


$574,938,293 


$483,219,157 


$10,683,024 


$0 


$2,309,136 


$7,323,191 


$1,330,224,061 


$513,004,856 


$22,218,156 


$98,147,150 


$1,487,411,634 


$25,179,471 


$7,330,632 


$0 


$6,471,973,020 


t.wwini 


%  PBSA 


Dollars  Actions 


9.71% 


13.26% 


46.81% 


0.00% 


21.31% 


3.45% 


12.14% 


0.00% 


24.65% 


11.47% 


63.80% 


70.79% 


30.86% 


0.00% 


11.61% 


40.98% 


37.35% 


59.86% 


50.50% 


41.56% 


94.74% 


30.20% 


67.24% 


0.00% 


29.01% 


5.77% 


18.36% 


45.98% 


0.00% 


20.00% 


12.50% 


42.86% 


0.00% 


23.73% 


1 1 .33% 


75.42% 


68.54% 


27.38% 


0.00% 


14.63% 


63.64% 


40.34% 


60.55% 


48.81  % 


31.85% 


79.66% 


42.67% 


0.00% 


27.90% 


%  Dollars  of  Total 


42.06% 


2.21% 


3.87% 


0.04% 


0.38% 


0.07% 


0.22% 


0.00% 


8.26% 


1 .83% 


4.04% 


3.06% 


0.16% 


0.00% 


0.09% 


0.08% 


15.96% 


3.84% 


0.20% 


1 .06% 


7.04% 


0.37% 


0.05% 


5.11% 


100.00% 


C5  -  Type  of  Contract 


Blank 


A  -  Firm  Fixed  Price  Redeterm 


J  -  Firm  Fixed  Price 


K  -  Fixed  Price  Econ  Price  Adj 


L  -  Fixed  Price  Incentive 


M  -  Fixed  Price  Award  Fee 


R  -  Cost  Plus  Award  Fee 


S  -  Cost  Contract 


T  -  Cost  Sharing 


U  -  Cost  Plus  Fixed  Fee 


V  -  Cost  Plus  Incentive  Fee 


Y  -  Time  and  Materials 


Z  -  Labor  Hour 


Total 


Total  T 

Dollars 

$1,084,860,282 

\mmm\ 

$57,239,870 

1  15  1 

$8,730,084,054 

$85,923,869 

$799,491,111 

$265,178,410 

$6,287,966,577 

$651,836,670 

$24,691,965 

51 

$2,289,395,582 

3790 

$413,727,372 

\m&m\ 

$1,465,221,100 

IKS! 

$155,220,346 

— 

$22,310,837,208 

FY  2002 


PBSA 


Dollars 


$498,603,796 


$1,754,112 


$3,125,975,698 


$47,077,487 


$366,573,315 


$160,831,283 


$1,634,430,280 


$22,087,011 


$197,104 


$241,046,273 


$67,516,249 


$263,380,346 


$42,500,066 


$6,471,973,020 


HWMlKli 


%  PBSA 


Dollars  Actions 


45.96% 


3.06% 


35.81% 


54.79% 


45.85% 


60.65% 


25.99% 


3.39% 


0.80% 


10.53% 


16.32% 


17.98% 


27.38% 


29.01% 


44.53% 


20.00% 


34.92% 


40.65% 


24.68% 


50.79% 


31.17% 


4.68% 


1 .96% 


6.33% 


14.10% 


26.58% 


27.63% 


27.90% 


%  Dollars  by  Total 


4.86% 


0.26% 


39.13% 


0.39% 


3.58% 


1.19% 


28.18% 


2.92% 


0.11% 


10.26% 


1 .85% 


6.57% 


0.70% 


100.00% 


D1A  -  Type  of  Entity 

Dollars 

Blank 

$1,084,860,282 

A  -  Small  Disadvantaged  in  U.S. 

$1,360,607,365 

B  -  Other  Small  in  U.S. 

$1,803,197,694 

C  -  Large  in  U.S. 

$15,521,075,218 

D  -  JWOD  Nonprofit  Agency 

$152,352,594 

F  -  Hospital 

$499,828 

L  -  Foreign  Concern/Entity 

$402,719,220 

M  -  Domestic  Firm  Outside  U.S. 

$580,988,101 

T  -  Historically  Black  College 

$1,763,791 

U  -  Minority  Institution 

$17,356,716 

V  -  Other  Educational 

$516,808,652 

Z  -  Other  Nonprofit 

$868,607,747 

Total 

$22,310,837,208 

|:rii] 

m 


FY  2002 


PBSA 


Dollars 


$498,603,796 


$425,063,809 


$528,897,624 


$4,326,819,042 


$77,507,811 


$499,828 


$132,047,363 


$241,634,219 


$287,447 


$274,592 


$22,292,128 


$218,045,361 


$6,471,973,020 


%  PBSA 


Dollars  Actions 


45.96% 


31.24% 


29.33% 


27.88% 


50.87% 


44.53% 


24.65% 


24.08% 


26.00% 


54.15% 


■  HH»IIHaifcllHWHHa| 


32.79% 


41 .59% 


16.30% 


1 .58% 


4.31% 


25.10% 


29.01% 


30.84% 


43.77% 


12.50% 


3.03% 


6.09% 


38.46% 


27.90% 


%  Dollars  by  Total 


4.86% 


6.10% 


8.08% 


69.57% 


0.68% 


0.00% 


1.81% 


2.60% 


0.01% 


0.08% 


2.32% 


3.89% 


100.00% 
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Appendix  E:  Summary  of  Percentage  of  Actions  and  Dollars  PBSA  for  Service 
Category,  Type  of  Contract,  and  Type  of  Entity  for  Fiscal  Year  2003 


Total 


Dollars 


$9,575,552,634 


$635,771,064 


$1,000,332,031 


$17,874,639 


$49,870,022 


$18,334,219 


$1,801,001,078 


$0 


$2,492,334,950 


$462,965,382 


$956,229,751 


$745,981,152 


$30,266,779 


$0 


$17,695,575 


$25,675,207 


$4,247,421,424 


$1,360,555,553 


$45,953,984 


$298,889,044 


$2,699,981,982 


$45,448,357 


$20,448,343 


$1,963,050,974 


$28,511,634,144 


KWtfJlKlI 


FY  2003 


PBSA 


Dollars 


$726,215,589 


$186,191,421 


$512,450,598 


$7,076,060 


$7,885,775 


$0 


$16,121,674 


$0 


$615,362,382 


$132,624,877 


$673,143,323 


$554,254,254 


$5,471,967 


$0 


$10,290,523 


$20,859,248 


$1,782,642,691 


$1,047,016,864 


$12,823,609 


$138,853,352 


$2,623,676,457 


$28,455,510 


$2,024,638 


$95,928,147 


$7,663,018,554 


jEjESBI 


%  PBSA 


Dollars  Actions 


7.58%  3.70% 


29.29%  44.32% 


51.23%  48.86% 


0.00%  0.00% 


15.81%  14.43% 


0.00%  0.00% 


0.90%  38.89% 


0.00%  0.00% 


24.69%  25.43% 


28.65%  25.51% 


70.40%  68.77% 


74.30%  77.89% 


18.08%  25.00% 


0.00%  0.00% 


58.15%  43.59% 


81.24%  70.97% 


41.97%  44.14% 


76.96%  75.55% 


27.91%  30.49% 


46.46%  40.00% 


97.17%  91.55% 


62.61%  49.04% 


9.90%  18.75% 


4.89%  3.77% 


26.88%  33.73% 


%  Dollars  of  Total 


33.58% 


2.23% 


3.51% 


0.06% 


0.17% 


0.06% 


6.32% 


0.00% 


8.74% 


1 .62% 


3.35% 


2.62% 


0.11% 


0.00% 


0.06% 


0.09% 


14.90% 


4.77% 


0.16% 


1 .05% 


9.47% 


0.16% 


0.07% 


6.89% 


100.00% 


C5  -  Type  of  Contract 


Blank 


A  -  Firm  Fixed  Price  Redeterm 


J  -  Firm  Fixed  Price 


K  -  Fixed  Price  Econ  Price  Adj 


L  -  Fixed  Price  Incentive 


M  -  Fixed  Price  Award  Fee 


R  -  Cost  Plus  Award  Fee 


S  -  Cost  Contract 


T  -  Cost  Sharing 


U  -  Cost  Plus  Fixed  Fee 


V  -  Cost  Plus  Incentive  Fee 


Y  -  Time  and  Materials 


Z  -  Labor  Hour 


Total 


Total 


Dollars 


$1,549,124,081 


$1,282,309 


$10,816,371,305 


$105,365,465 


$1,004,835,015 


$311,762,857 


$7,718,613,781 


$1,020,315,505 


$19,603,510 


$2,673,019,694 


$1,327,641,370 


$1,872,012,961 


$91,686,291 


$28,511,634,144 


KWMlKlI 


FY  2003 


PBSA 


Dollars 


$759,373,566 


$0 


$4,506,576,657 


$64,154,948 


$437,987,462 


$200,655,058 


$2,402,707,218 


$58,632,040 


$287,210 


$177,020,743 


$177,422,181 


$381,820,710 


$32,731,166 


$9,199,368,959 


%  PBSA 


Dollars  Actions 


49.02% 


0.00% 


41.66% 


60.89% 


43.59% 


64.36% 


31.13% 


5.75% 


1 .47% 


6.62% 


13.36% 


20.40% 


35.70% 


32.27% 


49.74% 


0.00% 


41.53% 


48.91% 


38.46% 


46.54% 


38.88% 


12.11% 


2.56% 


4.36% 


47.18% 


29.68% 


31.88% 


33.73% 


%  Dollars  by  Total 


5.43% 


0.00% 


37.94% 


0.37% 


3.52% 


1 .09% 


27.07% 


3.58% 


0.07% 


9.38% 


4.66% 


6.57% 


0.32% 


100.00% 


D1A  -  Type  of  Entity 

Dollars 

Blank 

$1,549,124,081 

A  -  Small  Disadvantaged  in  U.S. 

$1,969,279,080 

B  -  Other  Small  in  U.S. 

$1,977,648,257 

C  -  Large  in  U.S. 

$19,455,047,911 

D  -  JWOD  Nonprofit  Agency 

$182,285,964 

F  -  Hospital 

$0 

L  -  Foreign  Concern/Entity 

$547,313,160 

M  -  Domestic  Firm  Outside  U.S. 

$1,136,008,192 

T  -  Historically  Black  College 

$7,857,520 

U  -  Minority  Institution 

$23,619,055 

V  -  Other  Educational 

$741,796,964 

Z  -  Other  Nonprofit 

$921,653,960 

Total 

$28,511,634,144 

■nnnnaj 


FY  2003 


PBSA 


Dollars 


$759,373,566 


$639,129,131 


$610,972,697 


$5,748,012,382 


$109,845,413 


$0 


$163,695,701 


$719,790,367 


$292,398 


$278,684 


$41,179,254 


$406,799,366 


$9,199,368,959 


%  PBSA 


Dollars  Actions 


49.02% 


32.45% 


30.89% 


29.55% 


60.26% 


0.00% 


29.91% 


63.36% 


3.72% 


1.18% 


5.55% 


44.14% 


32.27% 


49.74% 


25.85% 


24.82% 


35.06% 


66.24% 


0.00% 


28.81% 


64.41% 


10.00% 


3.70% 


8.97% 


30.10% 


33.73% 


%  Dollars  by  Total 


5.43% 


6.91% 


6.94% 


68.24% 


0.64% 


0.00% 


1 .92% 


3.98% 


0.03% 


0.08% 


2.60% 


3.23% 


100.00% 
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Appendix  F:  Summary  of  Percentage  of  Actions  and  Dollars  PBSA  for  Service 
Category,  Type  of  Contract,  and  Type  of  Entity  for  Fiscal  Year  2004 


Total 


Dollars 


$10,189,430,033 


$514,726,664 


$1,170,926,890 


$15,361,540 


$23,748,131 


$14,839,169 


$44,170,851 


$0 


$2,787,632,290 


$370,755,305 


$918,150,538 


$802,913,906 


$15,911,123 


$0 


$12,632,205 


$33,328,602 


$4,387,159,765 


$1,288,053,619 


$35,925,233 


$255,153,508 


$2,250,437,179 


$23,590,619 


$16,633,739 


$1,478,510,827 


$26,649,991,736 


KWtfJlKlI 


wmm 


FY  2004 


PBSA 


Dollars 


$1,055,063,639 


$173,605,864 


$485,624,303 


$5,624,880 


$7,619,569 


$0 


$4,628,299 


$0 


$438,504,910 


$103,485,248 


$309,530,630 


$351,133,681 


$1,651,236 


$0 


$2,082,612 


$11,528,181 


$1,495,270,742 


$902,777,998 


$7,637,980 


$85,236,097 


$2,182,552,287 


$9,953,067 


$1,451,198 


$28,056,133 


$7,663,018,554 


EBjEjli 


%  PBSA 


Dollars  Actions 


10.35% 


33.73% 


41.47% 


0.00% 


32.08% 


0.00% 


10.48% 


0.00% 


15.73% 


27.91% 


33.71% 


43.73% 


10.38% 


0.00% 


16.49% 


34.59% 


34.08% 


70.09% 


21 .26% 


33.41% 


96.98% 


42.19% 


8.72% 


1 .90% 


28.75% 


23.02% 


41.46% 


38.68% 


0.00% 


33.90% 


0.00% 


19.64% 


0.00% 


14.76% 


17.65% 


45.99% 


53.98% 


11.11% 


0.00% 


7.41% 


32.71% 


42.09% 


51.32% 


22.08% 


21.25% 


85.97% 


21.21% 


5.56% 


2.39% 


32.09% 


%  Dollars  of  Total 


38.23% 


1 .93% 


4.39% 


0.06% 


0.09% 


0.06% 


0.17% 


0.00% 


10.46% 


1 .39% 


3.45% 


3.01% 


0.06% 


0.00% 


0.05% 


0.13% 


16.46% 


4.83% 


0.13% 


0.96% 


8.44% 


0.09% 


0.06% 


5.55% 


100.00% 


C5  -  Type  of  Contract 


Blank 


A  -  Firm  Fixed  Price  Redeterm 


J  -  Firm  Fixed  Price 


K  -  Fixed  Price  Econ  Price  Adj 


L  -  Fixed  Price  Incentive 


M  -  Fixed  Price  Award  Fee 


R  -  Cost  Plus  Award  Fee 


S  -  Cost  Contract 


T  -  Cost  Sharing 


U  -  Cost  Plus  Fixed  Fee 


V  -  Cost  Plus  Incentive  Fee 


Y  -  Time  and  Materials 


Z  -  Labor  Hour 


Total 


Total 


Dollars 


$1,835,247,266 


$5,694,130 


$8,320,539,039 


$57,612,003 


$533,700,092 


$251,423,539 


$6,822,988,443 


$1,348,607,615 


$21,750,240 


$4,059,934,898 


$1,138,326,852 


$2,152,576,133 


$101,591,486 


$26,649,991,736 


wmsm 


FY  2004 


PBSA 


Dollars 


$774,622,087 


$584,248 


$3,551,485,090 


$40,581,918 


$98,393,381 


$138,131,128 


$1,455,474,240 


$265,036,550 


$3,261,066 


$710,685,413 


$263,135,801 


$332,187,028 


$29,440,604 


$7,663,018,554 


EjEjli 


%  PBSA 


Dollars  Actions 


42.21%  44.45% 


10.26%  25.00% 


42.68%  32.05% 


70.44%  51.85% 


18.44%  16.67% 


54.94%  50.96% 


21.33%  33.97% 


19.65%  22.50% 


14.99%  28.13% 


17.50%  28.49% 


23.12%  47.80% 


15.43%  22.92% 


28.98%  30.29% 


28.75%  32.09% 


%  Dollars  by  Total 


6.89% 


0.02% 


31.22% 


0.22% 


2.00% 


0.94% 


25.60% 


5.06% 


0.08% 


15.23% 


4.27% 


8.08% 


0.38% 


100.00% 


D1A  -  Type  of  Entity 

Dollars 

Blank 

$1,835,247,266 

A  -  Small  Disadvantaged  in  U.S. 

$1,708,032,459 

B  -  Other  Small  in  U.S. 

$2,222,974,939 

C  -  Large  in  U.S. 

$17,160,563,691 

D  -  JWOD  Nonprofit  Agency 

$185,737,008 

F  -  Hospital 

$0 

L  -  Foreign  Concern/Entity 

$569,502,895 

M  -  Domestic  Firm  Outside  U.S. 

$1,059,490,953 

T  -  Historically  Black  College 

$258,409 

U  -  Minority  Institution 

$29,367,435 

V  -  Other  Educational 

$858,241,084 

Z  -  Other  Nonprofit 

$1,020,575,597 

Total 

$26,649,991,736 

mgm 

l 


FY  2004 


PBSA 


Dollars 


$774,622,087 


$434,865,318 


$817,337,558 


$4,492,855,188 


$46,138,363 


$0 


$155,959,853 


$663,788,582 


$119,999 


$2,590,740 


$213,364,499 


$61,376,367 


$7,663,018,554 


%  PBSA 


Dollars  Actions 


42.21% 


25.46% 


36.77% 


26.18% 


24.84% 


0.00% 


27.39% 


62.65% 


46.44% 


8.82% 


24.86% 


6.01% 


28.75% 


44.45% 


19.95% 


32.77% 


32.38% 


32.51% 


0.00% 


31.00% 


50.38% 


50.00% 


19.57% 


18.18% 


39.27% 


32.09% 


%  Dollars  by  Total 


6.89% 


6.41% 


8.34% 


64.39% 


0.70% 


0.00% 


2.14% 


3.98% 


0.00% 


0.11% 


3.22% 


3.83% 


100.00% 
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Appendix  G:  Contingency  Tables 


PBSA 


Count 

Total  % 

Col  % 

Row  % 

N 

Y 

N 

54260 

22267 

76527 

70.71 

29.02 

99.73 

99.87 

99.38 

70.90 

29.10 

Y 

71 

140 

211 

0.09 

0.18 

0.27 

0.13 

0.62 

33.65 

66.35 

54331 

22407 

76738 

70.80 

29.20 

PBSA 


Count 

Total  % 

Col  % 

Row  % 

N 

Y 

? 

54287 

22309 

76596 

70.74 

29.07 

99.81 

99.92 

99.56 

70.87 

29.13 

A 

1 

16 

17 

0.00 

0.02 

0.02 

0.00 

0.07 

5.88 

94.12 

B 

10 

59 

69 

0.01 

0.08 

0.09 

0.02 

0.26 

14.49 

85.51 

C 

33 

23 

56 

0.04 

0.03 

0.07 

0.06 

0.10 

58.93 

41.07 

54331 

70.80 

22407 

29.20 

76738 
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PBSA 


Count 

Total  % 

Col  % 

Row  % 

N 

Y 

N 

54325 

22404 

76729 

70.79 

29.20 

99.99 

99.99 

99.99 

70.80 

29.20 

Y 

6 

3 

9 

0.01 

0.00 

0.01 

0.01 

0.01 

66.67 

33.33 

54331 

22407 

76738 

70.80 

29.20 
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PBSA 


Count 

Total  % 

Col  % 

Row  % 

N 

Y 

1 

352 

63 

415 

0.46 

0.08 

0.54 

0.65 

0.28 

84.82 

15.18 

3 

20675 

6191 

26866 

26.94 

8.07 

35.01 

38.05 

27.63 

76.96 

23.04 

4 

768 

110 

878 

1.00 

0.14 

1.14 

1.41 

0.49 

87.47 

12.53 

5 

26104 

11528 

37632 

34.02 

15.02 

49.04 

48.05 

51.45 

69.37 

30.63 

6 

4714 

3836 

8550 

6.14 

5.00 

11.14 

8.68 

17.12 

55.13 

44.87 

7 

99 

35 

134 

0.13 

0.05 

0.17 

0.18 

0.16 

73.88 

26.12 

8 

173 

193 

366 

0.23 

0.25 

0.48 

0.32 

0.86 

47.27 

52.73 

9 

1446 

451 

1897 

1.88 

0.59 

2.47 

2.66 

2.01 

76.23 

23.77 

54331 

22407 

76738 

70.80 

29.20 
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PBSA 


Count 

Total  % 

Col  % 

Row  % 

N 

Y 

? 

26046 

10195 

36241 

33.94 

13.29 

47.23 

47.94 

45.50 

71.87 

28.13 

A 

6368 

2478 

8846 

8.30 

3.23 

11.53 

11.72 

11.06 

71.99 

28.01 

B 

21174 

9552 

30726 

27.59 

12.45 

40.04 

38.97 

42.63 

68.91 

31.09 

C 

743 

182 

925 

0.97 

0.24 

1.21 

1.37 

0.81 

80.32 

19.68 

54331 

22407 

76738 

70.80 

29.20 
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PBSA 


Count 

Total  % 

Col  % 

Row  % 

N 

Y 

? 

48895 

16618 

65513 

63.72 

21.66 

85.37 

89.99 

74.16 

74.63 

25.37 

A 

4380 

915 

5295 

5.71 

1.19 

6.90 

8.06 

4.08 

82.72 

17.28 

B 

19 

12 

31 

0.02 

0.02 

0.04 

0.03 

0.05 

61.29 

38.71 

C 

451 

196 

647 

0.59 

0.26 

0.84 

0.83 

0.87 

69.71 

30.29 

D 

408 

521 

929 

0.53 

0.68 

1.21 

0.75 

2.33 

43.92 

56.08 

E 

178 

4145 

4323 

0.23 

5.40 

5.63 

0.33 

18.50 

4.12 

95.88 

54331 

22407 

76738 

70.80 

29.20 
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PBSA 


Count 

Total  % 

Col  % 

Row  % 

N 

Y 

? 

47869 

16801 

64670 

62.38 

21.89 

84.27 

88.11 

74.98 

74.02 

25.98 

A 

276 

160 

436 

0.36 

0.21 

0.57 

0.51 

0.71 

63.30 

36.70 

B 

383 

124 

507 

0.50 

0.16 

0.66 

0.70 

0.55 

75.54 

24.46 

C 

498 

269 

767 

0.65 

0.35 

1.00 

0.92 

1.20 

64.93 

35.07 

D 

5305 

5053 

10358 

6.91 

6.58 

13.50 

9.76 

22.55 

51.22 

48.78 

54331 

22407 

76738 

70.80 

29.20 

PBSA 


Count 

Total  % 

Col  % 

Row  % 

N 

Y 

? 

4757 

3859 

8616 

6.20 

5.03 

11.23 

8.76 

17.22 

55.21 

44.79 

A 

31636 

13350 

44986 

41.23 

17.40 

58.62 

58.23 

59.58 

70.32 

29.68 

B 

3773 

1167 

4940 

4.92 

1.52 

6.44 

6.94 

5.21 

76.38 

23.62 

N 

14165 

4031 

18196 

18.46 

5.25 

23.71 

26.07 

17.99 

77.85 

22.15 

54331 

22407 

76738 

70.80 

29.20 
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PBSA 


Count 

Total  % 

Col  % 

Row  % 

N 

Y 

? 

40166 

18376 

58542 

52.34 

23.95 

76.29 

73.93 

82.01 

68.61 

31.39 

A 

454 

112 

566 

0.59 

0.15 

0.74 

0.84 

0.50 

80.21 

19.79 

B 

202 

38 

240 

0.26 

0.05 

0.31 

0.37 

0.17 

84.17 

15.83 

C 

111 

31 

142 

0.14 

0.04 

0.19 

0.20 

0.14 

78.17 

21.83 

Z 

13398 

3850 

17248 

17.46 

5.02 

22.48 

24.66 

17.18 

77.68 

22.32 

54331 

22407 

76738 

70.80 

29.20 

113 


PBSA 


Count 

Total  % 

Col  % 

Row  % 

N 

Y 

? 

43 

23 

66 

0.06 

0.03 

0.09 

0.08 

0.10 

65.15 

34.85 

A 

41238 

18817 

60055 

53.74 

24.52 

78.26 

75.90 

83.98 

68.67 

31.33 

B 

4900 

1750 

6650 

6.39 

2.28 

8.67 

9.02 

7.81 

73.68 

26.32 

C 

1241 

257 

1498 

1.62 

0.33 

1.95 

2.28 

1.15 

82.84 

17.16 

D 

6909 

1560 

8469 

9.00 

2.03 

11.04 

12.72 

6.96 

81.58 

18.42 

54331 

22407 

76738 

70.80 

29.20 
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PBSA 


Count 

Total  % 

Col  % 

Row  % 

N 

Y 

? 

4757 

3859 

8616 

6.20 

5.03 

11.23 

8.76 

17.22 

55.21 

44.79 

A 

40 

6 

46 

0.05 

0.01 

0.06 

0.07 

0.03 

86.96 

13.04 

J 

22451 

11556 

34007 

29.26 

15.06 

44.32 

41.32 

51.57 

66.02 

33.98 

K 

263 

211 

474 

0.34 

0.27 

0.62 

0.48 

0.94 

55.49 

44.51 

L 

485 

157 

642 

0.63 

0.20 

0.84 

0.89 

0.70 

75.55 

24.45 

M 

371 

321 

692 

0.48 

0.42 

0.90 

0.68 

1.43 

53.61 

46.39 

R 

3817 

1843 

5660 

4.97 

2.40 

7.38 

7.03 

8.23 

67.44 

32.56 

S 

1991 

292 

2283 

2.59 

0.38 

2.98 

3.66 

1.30 

87.21 

12.79 

T 

158 

14 

172 

0.21 

0.02 

0.22 

0.29 

0.06 

91.86 

8.14 

U 

13366 

1841 

15207 

17.42 

2.40 

19.82 

24.60 

8.22 

87.89 

12.11 

V 

587 

315 

902 

0.76 

0.41 

1.18 

1.08 

1.41 

65.08 

34.92 

Y 

5507 

1716 

7223 

7.18 

2.24 

9.41 

10.14 

7.66 

76.24 

23.76 

Z 

538 

276 

814 

0.70 

0.36 

1.06 

0.99 

1.23 

66.09 

33.91 

54331 

22407 

76738 

70.80 

29.20 
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PBSA 


Count 

Total  % 

Col  % 

Row  % 

N 

Y 

? 

6993 

4491 

11484 

9.11 

5.85 

14.97 

12.87 

20.04 

60.89 

39.11 

A 

934 

167 

1101 

1.22 

0.22 

1.43 

1.72 

0.75 

84.83 

15.17 

B 

19574 

6173 

25747 

25.51 

8.04 

33.55 

36.03 

27.55 

76.02 

23.98 

C 

195 

46 

241 

0.25 

0.06 

0.31 

0.36 

0.21 

80.91 

19.09 

D 

3 

0 

3 

0.00 

0.00 

0.00 

0.01 

0.00 

100.00 

0.00 

E 

4984 

453 

5437 

6.49 

0.59 

7.09 

9.17 

2.02 

91.67 

8.33 

F 

481 

89 

570 

0.63 

0.12 

0.74 

0.89 

0.40 

84.39 

15.61 

G 

40 

23 

63 

0.05 

0.03 

0.08 

0.07 

0.10 

63.49 

36.51 

K 

8687 

3088 

11775 

11.32 

4.02 

15.34 

15.99 

13.78 

73.77 

26.23 

N 

12440 

7877 

20317 

16.21 

10.26 

26.48 

22.90 

35.15 

61.23 

38.77 

54331 

22407 

76738 

70.80 

29.20 
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PBSA 


Count 

Total  % 

Col  % 

Row  % 

N 

Y 

? 

5006 

3961 

8967 

6.52 

5.16 

11.69 

9.21 

17.68 

55.83 

44.17 

N 

45035 

14544 

59579 

58.69 

18.95 

77.64 

82.89 

64.91 

75.59 

24.41 

Y 

4290 

3902 

8192 

5.59 

5.08 

10.68 

7.90 

17.41 

52.37 

47.63 

54331 

22407 

76738 

70.80 

29.20 
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PBSA 


Count 

Total  % 

Col  % 

Row  % 

N 

Y 

? 

4757 

3859 

8616 

6.20 

5.03 

11.23 

8.76 

17.22 

55.21 

44.79 

A 

8832 

2599 

11431 

11.51 

3.39 

14.90 

16.26 

11.60 

77.26 

22.74 

B 

9871 

3219 

13090 

12.86 

4.19 

17.06 

18.17 

14.37 

75.41 

24.59 

C 

25092 

10157 

35249 

32.70 

13.24 

45.93 

46.18 

45.33 

71.18 

28.82 

D 

422 

399 

821 

0.55 

0.52 

1.07 

0.78 

1.78 

51.40 

48.60 

F 

0 

4 

4 

0.00 

0.01 

0.01 

0.00 

0.02 

0.00 

100.00 

L 

2159 

860 

3019 

2.81 

1.12 

3.93 

3.97 

3.84 

71.51 

28.49 

M 

735 

844 

1579 

0.96 

1.10 

2.06 

1.35 

3.77 

46.55 

53.45 

T 

23 

3 

26 

0.03 

0.00 

0.03 

0.04 

0.01 

88.46 

11.54 

U 

109 

13 

122 

0.14 

0.02 

0.16 

0.20 

0.06 

89.34 

10.66 

V 

1759 

192 

1951 

2.29 

0.25 

2.54 

3.24 

0.86 

90.16 

9.84 

Z 

572 

258 

830 

0.75 

0.34 

1.08 

1.05 

1.15 

68.92 

31.08 

54331 

22407 

76738 

70.80 

29.20 
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PBSA 


Count 

Total  % 

Col  % 

Row  % 

N 

Y 

? 

4757 

3859 

8616 

6.20 

5.03 

11.23 

8.76 

17.22 

55.21 

44.79 

N 

46464 

17300 

63764 

60.55 

22.54 

83.09 

85.52 

77.21 

72.87 

27.13 

U 

702 

362 

1064 

0.91 

0.47 

1.39 

1.29 

1.62 

65.98 

34.02 

Y 

2408 

886 

3294 

3.14 

1.15 

4.29 

4.43 

3.95 

73.10 

26.90 

54331 

22407 

76738 

70.80 

29.20 

PBSA 


Count 

Total  % 

Col  % 

Row  % 

N 

Y 

? 

35628 

16589 

52217 

46.43 

21.62 

68.05 

65.58 

74.03 

68.23 

31.77 

N 

16766 

5354 

22120 

21.85 

6.98 

28.83 

30.86 

23.89 

75.80 

24.20 

Y 

1937 

464 

2401 

2.52 

0.60 

3.13 

3.57 

2.07 

80.67 

19.33 

54331 

22407 

76738 

70.80 

29.20 
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PBSA 


Count 

Total  % 

Col  % 

Row  % 

N 

Y 

? 

45530 

19856 

65386 

59.33 

25.88 

85.21 

83.80 

88.62 

69.63 

30.37 

A 

385 

109 

494 

0.50 

0.14 

0.64 

0.71 

0.49 

77.94 

22.06 

B 

868 

358 

1226 

1.13 

0.47 

1.60 

1.60 

1.60 

70.80 

29.20 

C 

1587 

595 

2182 

2.07 

0.78 

2.84 

2.92 

2.66 

72.73 

27.27 

D 

1882 

441 

2323 

2.45 

0.57 

3.03 

3.46 

1.97 

81.02 

18.98 

E 

2598 

663 

3261 

3.39 

0.86 

4.25 

4.78 

2.96 

79.67 

20.33 

F 

500 

105 

605 

0.65 

0.14 

0.79 

0.92 

0.47 

82.64 

17.36 

Z 

981 

280 

1261 

1.28 

0.36 

1.64 

1.81 

1.25 

77.80 

22.20 

54331 

22407 

76738 

70.80 

29.20 
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PBSA 


Count 

Total  % 

Col  % 

Row  % 

N 

Y 

? 

4757 

3859 

8616 

6.20 

5.03 

11.23 

8.76 

17.22 

55.21 

44.79 

A 

37237 

14272 

51509 

48.52 

18.60 

67.12 

68.54 

63.69 

72.29 

27.71 

B 

5229 

2361 

7590 

6.81 

3.08 

9.89 

9.62 

10.54 

68.89 

31.11 

C 

279 

71 

350 

0.36 

0.09 

0.46 

0.51 

0.32 

79.71 

20.29 

D 

6151 

1664 

7815 

8.02 

2.17 

10.18 

11.32 

7.43 

78.71 

21.29 

E 

214 

56 

270 

0.28 

0.07 

0.35 

0.39 

0.25 

79.26 

20.74 

F 

73 

12 

85 

0.10 

0.02 

0.11 

0.13 

0.05 

85.88 

14.12 

K 

298 

90 

388 

0.39 

0.12 

0.51 

0.55 

0.40 

76.80 

23.20 

L 

93 

22 

115 

0.12 

0.03 

0.15 

0.17 

0.10 

80.87 

19.13 

54331 

22407 

76738 

70.80 

29.20 
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PBSA 


Count 

Total  % 

Col  % 

Row  % 

N 

Y 

? 

4757 

3859 

8616 

6.20 

5.03 

11.23 

8.76 

17.22 

55.21 

44.79 

A 

49556 

18544 

68100 

64.58 

24.17 

88.74 

91.21 

82.76 

72.77 

27.23 

B 

3 

1 

4 

0.00 

0.00 

0.01 

0.01 

0.00 

75.00 

25.00 

D 

15 

3 

18 

0.02 

0.00 

0.02 

0.03 

0.01 

83.33 

16.67 

54331 

22407 

76738 

70.80 

29.20 
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PBSA 


Count 

Total  % 

Col  % 

Row  % 

N 

Y 

? 

5055 

4001 

9056 

6.59 

5.21 

11.80 

9.30 

17.86 

55.82 

44.18 

A 

46746 

17932 

64678 

60.92 

23.37 

84.28 

86.04 

80.03 

72.27 

27.73 

B 

47 

8 

55 

0.06 

0.01 

0.07 

0.09 

0.04 

85.45 

14.55 

C 

2362 

449 

2811 

3.08 

0.59 

3.66 

4.35 

2.00 

84.03 

15.97 

D 

121 

17 

138 

0.16 

0.02 

0.18 

0.22 

0.08 

87.68 

12.32 

54331 

22407 

76738 

70.80 

29.20 

Any  row  with  an  “?”  indicates  that  no  code  was  selected  for  that  variable  for  that 
particular  action. 
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Appendix  H:  Parameter  Estimates  for  Logistic  Regression  Model  without  Unstable 

or  Biased  Variables. 


Parameter  Estimates 


Term 

Intercept 

B1C  -  Bundled  Contract  [N] 

B13B  -  Type  of  Indefinite-Delivery  Contract  [A] 

B13B  -  Type  of  Indefinite-Delivery  Contract  [B] 

B13B  -  Type  of  Indefinite-Delivery  Contract  [C] 

B13E  -  Multiple  Award  Contract  Fair  Opportunity  [A] 
B13E  -  Multiple  Award  Contract  Fair  Opportunity  [B] 
B13E  -  Multiple  Award  Contract  Fair  Opportunity  [C] 
B13E  -  Multiple  Award  Contract  Fair  Opportunity  [D] 
B13E  -  Multiple  Award  Contract  Fair  Opportunity  [E] 
B13F  -  Indefinite-Delivery  Contract  Use  [A] 

B13F  -  Indefinite-Delivery  Contract  Use  [B] 

B13F  -  Indefinite-Delivery  Contract  Use  [C] 

B13F  -  Indefinite-Delivery  Contract  Use  [D] 

Cl 4  -  Commercial  Items  [N] 

Cl 4  -  Commercial  Items  [Y] 

D1C  -  HUBZone  Representation  [N] 

D1C  -  HUBZone  Representation  [Y] 

DID  -  Ethnic  Group  [A] 

DID  -  Ethnic  Group  [B] 

DID  -  Ethnic  Group  [C] 

DID  -  Ethnic  Group  [D] 

DID  -  Ethnic  Group  [E] 

DID  -  Ethnic  Group  [F] 

DID  -  Ethnic  Group  [Z] 

D7  -  SBIR  Program  [A] 

D7  -  SBIR  Program  [B] 

D7  -  SBIR  Program  [C] 

D7  -  SBIR  Program  [D] 

For  log  odds  of  FI/M 


Estimate 

Std  Error 

ChiSquare 

Prob>ChiSq 

1.15945266 

0.1493497 

60.27 

<.0001 

-1.0217948 

0.0742649 

189.30 

<.0001 

0.11027158 

0.029991 

13.52 

0.0002 

0.00593461 

0.0269594 

0.05 

0.8258 

-0.2293267 

0.0658999 

12.11 

0.0005 

-1.4259021 

0.0733505 

377.90 

<.0001 

-0.3498394 

0.3140916 

1.24 

0.2654 

-0.6077224 

0.0975302 

38.83 

<.0001 

0.37374773 

0.0867607 

18.56 

<.0001 

3.24672216 

0.0928443 

1222.9 

<.0001 

0.0541297 

0.089726 

0.36 

0.5463 

-0.2464587 

0.0939629 

6.88 

0.0087 

0.36786891 

0.0720396 

26.08 

<.0001 

0.07654513 

0.0420865 

3.31 

0.0689 

-0.4662438 

0.072445 

41.42 

<.0001 

0.94984336 

0.0739473 

164.99 

<.0001 

0.11716142 

0.0222081 

27.83 

<.0001 

-0.2752741 

0.0397144 

48.04 

<.0001 

0.12115866 

0.1018073 

1.42 

0.2340 

0.32562806 

0.0639912 

25.89 

<.0001 

0.20367995 

0.0518348 

15.44 

<.0001 

-0.259404 

0.0562419 

21.27 

<.0001 

0.07389066 

0.0475593 

2.41 

0.1203 

-0.5987701 

0.1075247 

31.01 

<.0001 

-0.000672 

0.0673468 

0.00 

0.9920 

-0.0412664 

0.1044959 

0.16 

0.6929 

-0.3835844 

0.3166887 

1.47 

0.2258 

-0.2049099 

0.1111629 

3.40 

0.0653 

-0.5647882 

0.2275421 

6.16 

0.0131 
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